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Preface 


This  study  applied  Intel’s  artificial  neural  network  hardware,  the  80170NX 
chip  known  also  as  the  Electronically  Trainable  Analog  Neural  Network  (ETANN), 
to  classify  radar  emitter  systems.  This  research  study  concentrated  on  comparing 
simulator  results  to  that  of  actual  hardware  implementation  for  a  specific  application 
using  collected  data. 
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Abstract 

This  study  investigated  classification  of  30  radar  emitters  with  16  signal  fea¬ 
tures  using  Intel’s  80170NX  chip,  the  Electronically  TVainable  Analog  Neural  Net¬ 
work  (ETANN).  Software  tools  were  developed  to  characterize  the  ETANN  sigmoidal 
transfer  function  for  use  in  a  custom  simulator,  known  as  Neural  Graphics.  Neural 
Graphics  operates  on  a  Silicon  Graphics  workstation.  The  Intel  Neural  Network 
Training  System  simulators  were  used  in  early  experiments,  but  were  found  to  be 
inefficient  in  training  on  data  used  in  this  research.  Using  a  modified  Neural  Graph¬ 
ics  simulator,  single  chip  and  multi-chip  experiments  were  performed  to  provide 
benchmark  results  prior  to  performing  chip-in-loop  training.  By  maximizing  ofif-chip 
training  accuracy,  the  need  for  on-chip  training  is  minimized  and  therefore  the  de¬ 
vice  life  is  prolonged.  Several  single  chip  and  multi-chip  configurations  were  tried; 
the  final  architecture  which  produced  the  maximum  on-chip  classification  accuracy 
was  a  hierarchical  network.  The  maximum  on-chip  classification  accuracy  for  a  sin¬ 
gle  chip  implementation  of  30  classes  without  chip- in-loop  training  was  83  percent. 
Again  without  chip-in-loop  training,  the  maximum  on-chip  classification  accuracy 
for  a  hierarchical  configuration  with  the  30-class  problem  was  87  percent. 
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ETANN  HARDWARE  IMPLEMENTATION  FOR 
RADAR  EMITTER  IDENTIFICATION 


L  INTRODUCTION 


1.1  General  Issue 

Electronic  warfare  continues  to  he  a  primary  focus  of  Air  Force  research  and 
development.  From  the  simple  concept  of  using  radar  detectors  to  warn  pilots  of 
possible  hostile  threats  has  evolved  the  idea  of  identifying  those  threats,  so  pilots 
may  gain  the  advantage  by  taking  active,  evasive  action.  Artificial  Neural  Networks 
(ANNs),  also  referred  to  as  neural  networks,  now  make  it  possible  to  explore  old 
ideas  of  automating  emitter  identification  in  cockpits.  Because  aircraft  have  limited 
space  and  computing  capacity,  ANNs  may  be  the  ideal  solution  to  radar  emitter 
identification.  This  discussion  leads  precisely  to  what  this  thesis  addresses,  solving 
a  radar  emitter  identification  problem  using  ANN  hardware. 

1.2  Background 

An  ANN  is  a  model  that  simulates  a  biological  neural  network  system;  it 
may  model  brain  processes  or  brain  capabilities.  The  word  neural  was  derived  from 
neuron.  A  neuron  is  a  nerve  cell  that  is  used  to  process,  store,  retrieve,  and  elec- 
trochemically  manipulate  information  received  from  the  cell  body  or  input  channels, 
called  dendrites.  The  human  body  contains  tens  of  billions  of  neurons,  each  linked  to 
potentially  thousands  of  others  through  a  biological  network.  Each  neuron  features 
several  inputs,  but  only  one  output.  To  an  output,  the  neuron  sums  the 

inputs  from  the  dendrites  at  a  common  point  and  then  compares  the  sum  against 
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a  threshold  value.  If  the  threshold  value  is  exceeded,  the  neuron  activates  a  pulse 
voltage  in  the  output.  Similarly,  an  artificial  neuron  may  have  many  inputs,  but  only 
one  output.  Figure  1  illustrates  a  biological  neuron,  and  Figure  2  illustrates  an  arti¬ 
ficial  neuron,  which  is  also  referred  to  as  a  single  perception  proposed  by  Rosenblatt 
in  1959  (29).  The  only  true  similarity  between  a  biological  and  artificial  neuron  is 
the  input/output  structure  and  their  activation  nature.  A  parallel  combination  cf 
neurons  constitutes  a  network  layer,  and  one  or  more  layers  defines  a  neural  network. 
Two  or  more  layers  is  also  referred  to  as  a  multilayer  perceptron  network.  Moreover, 
the  ANN  network  n  ^y  have  more  than  one  output,  depending  on  how  many  decision 
functions  are  associated  with  the  ANN  output  (27:1-15). 

The  hardware  implementation  of  a  complex  ANN  model  makes  use  of  a  mas¬ 
sively  parallel  interconnected  network  of  perceptrons  (the  artificial  equivalent  of  a 
biological  neural  network),  which  interacts  in  a  way  similar  to  a  biological  nervt  us 
system.  The  ANN  accomplishes  tasks  by  modeling  a  biological  system;  specifically, 
the  ANN  attempts  to  assign  an  unknown  pattern  from  the  environment  into  one 
of  a  set  of  selected  classes  (supervised  method)  by  emulating  the  biological  neural 
network. 

Engineers  and  scientists  are  applying  the  ANN  to  such  tasks  as  automatic  tar¬ 
get  recognition.  As  described  by  Ruck  (30),  the  process  of  automatic  target  recog¬ 
nition  consists  of  three  stages:  segmentation,  feature  extraction,  and  classification. 
Segmentation  is  the  operation  of  isola/ting  the  target  of  interest  from  its  environment, 
whereas  feature  extraction  involves  processing  the  data  to  compute  the  features  that 
allow  discrimination  among  different  target  classes.  The  classification  stage  assigns 
each  input  target  to  a  class  based  on  its  features. 

It  has  been  demonstrated  that  neural  networks  can  be  used  to  design  radar 
classification  systems  on  the  order  of  30  classes  with  high  accuracy.  Cameron  (5) 
demonstrated  that  using  a  hierarchical  approach  to  classify  30  classes  results  in 
improved  system  accuracy  over  that  of  a  straight  forward  application  of  one  network. 
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Feet 


Figure  1.  Biological  Neuron  (28:19) 


Figure  2.  Artificial  Neuron  or  Single  Perceptron  (28:47) 
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Using  the  hierarchical  approach,  Capt  Cameron  succeeded  in  achieving  a  96.5  percent 
accuracy  rate  of  classifying  the  radar  emitter  patterns  correctly  (5:52).  A  discussion 
on  the  hierarchical  approach  used  in  this  research  is  covered  in  Chapter  4. 

1.3  Problem  Statement 

The  research  problem  is  to  implement  a  simulated  neural  network  from  (5) 
in  hardware  and  to  achieve  the  accuracy  results  recorded  from  simulation.  The 
hardware  used  in  these  experiments  is  built  by  the  Intel  Corporation.  The  primary 
hardware  device  is  the  80170NX  chip,  an  Electronically  Trainable  Analog  Neural 
Network  (ETANN).  In  addition  to  the  ETANN  chip,  Intel  built  the  Personal  Com¬ 
puter  (PC)  interface  hardware  and  software  that  allows  the  designer  to  program  the 
ETANN  chip  after  simulator  training  (13:1-3). 

Originally,  the  principal  investigator  (5)  was  given  32  classes  of  radar  signals  to 
implement  in  an  artificial  neural  network  simulator.  Two  of  the  classes  had  too  few  of 
vectors  to  train  effectively;  therefore,  they  were  excluded  from  further  experiments, 
and  thus  the  problem  was  further  defined  as  30-classes.  Then  the  Neural  Graphics 
simulator,  which  was  coded  in  the  C  computer  language  at  AFIT  (37),  was  used  to 
train  on  the  radar  signal  emitter  patterns.  The  back  propagation  training  paradigm 
was  used  in  Neural  Graphics. 

Little  has  been  said  about  the  data  because  the  focus  of  this  research  is  on 
hardware  implementation.  The  data  used  for  trmning  and  testing  the  hardware  is 
real,  collected  data,  intercepted  and  recorded  by  the  Georgia  Technical  Research 
Institute  (GTRl).  This  data  was  used  in  previous  experiments  (5). 

The  features  of  this  data  were  extracted  by  GTRI  and  transformed  into  samples 
of  data  vectors  in  American  Standard  Code  for  Information  Interchange  (ASCII) 
format.  Each  vector  contains  16  elements  representing  16  features  used  for  pattern 
classification.  The  vector  elements  are  measured  features  of  the  intercepted  radar 
signals. 
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1.4  Research  Objectives 

The  primary  research  objective  is  to  implement  an  artificial  neural  network  in 
Intel’s  ETANN  hardware  to  classify  the  GTRI  radar  emitter  data.  In  addition  to 
the  primary  research  objective,  the  following  sub-objectives  will  be  accomplished; 

1.  Minimize  the  processing  time  and  maximize  the  classification  accuracy.  This 
sub-objective  will  require  setting  up  experiments  to  prune  the  network  of  bad 
data  and  to  consider  chip-in-loop  training,  which  requires  using  the  ETANN 
chip  to  train  in  real  time.  Minimizing  the  processing  time  implies  using  as  few 
processing  layers  as  possible. 

2.  Develop  a  technique  to  load  training  weights  from  the  Neural  Graphics  simu¬ 
lator  (37)  to  Intel’s  Neural  Network  Training  System  (iNNTS).  This  technique 
will  require  transfer  of  the  normalized  data  as  well  as  the  weights. 

3.  Estimate  hardware  processing  time  to  make  a  classification.  The  processing 
time  in  hardware  is  important  information  that  will  indicate  how  close  to  real 
time  the  processor  functions. 

4.  Compare  resolution  differences  between  the  hardware  device  and  simulator. 
Resolution  is  important  because  information  is  usually  lost  when  going  from 
simulator  to  hardware,  which  may  affect  network  performance  in  classifying 
the  radar  emitter  feature  data. 

5.  Calculate  an  implementation  loss  factor.  The  loss  factor  is  a  loss  associated 
with  hardware  implementation  when  going  from  theoretical  to  actual  results. 
In  other  words,  the  loss  factor  is  the  difference  between  theoretical  (software 
simulation)  and  actual  results  (hardware  implementation). 

1.5  Scope 

This  research  focuses  on  a  single  application  of  classifying  30  radar  emitters  in 
ANN  hardware  based  on  discrimination  of  feature  vectors.  To  narrow  the  scope,  this 
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research  is  exclusively  limited  to  using  Intel’s  ETANN  chip  as  the  neural  network 
hardware  device.  The  simulators  used  in  this  research  are  AFIT’s  Neural  Graphics 
(37)  and  iBrainMaker  (3).  The  data  used  in  training  were  limited  to  a  fixed  number 
of  feature  vectors  (a  pool  of  8,469  vectore). 

1.6  Chapter  Outlines 

In  chapter  2,  a  literature  review  was  accomplished  to  look  at  all  the  information 
necessary  for  understanding  the  radar  emitter  classification  problem  and  hardware 
solution  approaches.  Chapter  3  presents  a  detail  map  of  the  methodology  used  in 
the  forming  of  a  solution  to  the  research  problem.  Chapter  4  presents  the  findings 
of  the  research  problem  and  sub-objectives.  Finally,  Chapter  5  presents  conclusions 
and  recommendations. 

1.7  Summary 

The  purpose  of  this  research  is  to  solve  a  30-class  radar  emitter  identification 
problem  using  Intel’s  neural  network  hardware  device,  the  ETANN  chip.  In  addition, 
a  series  of  tests  and  comparisons  with  simulation  results  were  performed  in  an  effort 
to  establish  a  theoretical  versus  practical  implementation  loss  factor  for  this  network. 
This  loss  factor  is  a  measure  by  which  future  simulations  can  be  adjusted  for  accuracy. 
Two  simulators  were  used  for  comparison  of  training  results.  The  degree  of  success 
of  this  research  will  help  determine  future  Air  Force  developmental  work  in  neural 
network  hardware. 
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II.  LITERATURE  REVIEW 


2.1  Introduction 

This  chapter  reviews  literature  pertinent 
classification  problem  and  solution  approaches 
in  this  chapter: 

•  Radar  Emitter  Classification 

•  Neural  Network  Hardware 

2.2  Radar  Emitter  Classification 

Artificial  neural  networks  are  rapidly  becoming  the  favorite  algorithm  in  emit¬ 
ter  identification.  The  speed  at  which  neural  networks  operate  and  the  robustness 
of  the  network  architecture  make  them  attractive  to  electronic  warfare  engineers. 
This  review  of  radar  emitter  classification  is  intended  to  document  a  baseline  of 
parameters  achieved  in  neural  network  simulations,  so  that  the  researcher  can  com¬ 
pare  hardware  implementation  results  of  the  Electronically  Trainable  Analog  Neural 
Network  (ETANN)  to  that  of  simulations  in  general. 

Radar  emitter  classification  results  until  now  have  only  been  documented  based 
on  simulation  results.  Thus  far,  the  best  simulation  results  of  a  large-class  emitter 
identification  classifier  were  achieved  by  Capt  Cameron  at  the  Air  Force  Institute 
of  Technology  (5).  His  best  test  results  for  a  30-class,  16-feature  problem  was  a  96 
percent  classification  accuracy  using  a  hierarchical  network  approach  (5:52). 

In  a  medium-class  problem,  Capt  Zahimiak  at  the  Air  Force  Institute  of  Tech¬ 
nology  used  a  kernel  classifier  (radial  basis  function)  to  classify  radar  emitters.  His 
network  was  configured  for  six  features  on  the  input  and  ten  classes  on  the  output. 
Capt  Zahimiak  achieved  an  86  percent  classification  accuracy  (43:5-17). 


to  understanding  the  radar  emitter 
Below  are  the  two  topics  discussed 
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Willson  describes  the  use  of  an  artificial  neural  network  (ANN)  for  classify¬ 
ing  radar  signals  collected  by  a  passive  receiver  (42).  He  used  an  ANN  employing 
a  multilayer  perceptron  network  using  back  propagation  training  to  classify  radar 
emitters.  Willson’s  network  consisted  of  six  input  features  and  fourteen  classes  on 
the  output.  Using  monopulse  radar  data  as  features  for  the  input  of  the  network, 
Willson  compared  the  ANN  employing  a  multilayer  perceptron  to  a  template-based 
technique  and  a  K-nearest  neighbor  classifier.  He  achieved  an  average  of  94  percent 
correct  classification  using  an  ANN,  an  average  of  86  percent  correct  classification 
using  a  K-nearest  neighbor  algorithm,  and  an  average  of  74  percent  correct  classifi¬ 
cation  using  a  template  match. 

At  the  Georgia  Tech  Research  Institute,  Howitt  used  a  standard  feed-forward 
network  with  three  input  features  (RF,  PRI,  and  PW),  eight  hidden  nodes,  and 
ten  emitter  classes  to  test  the  performance  of  an  ANN  using  the  back  propagation 
training  algorithm  (11:213).  Howitt ’s  simulator  results  for  four  independent  tests 
using  testing  data  sets  (data  not  shown  to  the  networks  before)  developed  for  each 
test  showed  an  average  correct  classification  of  99.9  percent. 

Standard  feed-forward  networks  suffer  from  several  drawbacks.  First,  network 
training  size  is  directly  related  to  the  size  of  the  network.  In  other  words,  as  the 
number  of  emitter  classes  increase,  the  rate  at  which  the  network  converges  decreases 
(slower  convergence).  Second,  when  a  network  is  modified  by  adding  or  deleting 
emitter  classes,  it  has  to  be  retrained  (the  entire  network  of  all  classes).  To  mitigate 
these  drawbacks,  Howitt  developed  an  alternate  method  to  the  standard  feed-forward 
network,  called  the  piecewise  training  network. 

Howitt  divided  the  standard  feed-forward  network  described  above  into  two 
networks,  each  with  three  input  features  as  before,  three  hidden  nodes  instead  of 
eight,  and  five  output  classes  instead  of  ten.  The  networks  were  trained  individually 
and  “glued”  into  one  final  network  with  two  additional  hidden  nodes.  After  com¬ 
bining  this  network,  an  additional  training  phase  with  the  entire  training  data  set 
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Table  1.  Radar  Emitter  Classification 


Classes 

Features 

Type 

Result 

Researcher 

10 

MLP 

99% 

Howitt 

10 

RBF 

86% 

Zahimiak 

16 

MLP 

94% 

Willson 

30 

16 

MLP 

96% 

Cameron 

is  required  (11:214).  For  the  four  simulator  tests  performed,  Howitt  found  results 
almost  identical  to  the  standard  feed-forward  network  (99  percent  average  correct 
classification). 

According  to  Howitt,  the  advantage  of  using  the  piecewise  network  in  this 
example  is  that  the  number  of  interconnections  were  decreased  from  104  to  74,  rep¬ 
resenting  a  30  percent  reduction  in  the  number  of  weights.  Fewer  weights  means  less 
training  time  required.  Moreover,  dividing  a  large  network  into  sub-networks  im¬ 
proves  convergence  time  (for  separable  classes).  Last  and  most  importantly,  adding 
or  deleting  classes  can  be  done  more  efficiently.  For  example,  the  entire  network  does 
not  require  retraining  when  adding  or  deleting  one  class.  By  retraining  only  a  sub¬ 
network  in  which  one  modifies  an  emitter  class  and  then  retraining  with  the  “glue” 
neurons  in  the  entire  network,  one  saves  having  to  retrain  the  other  sub-networks. 
This  method  could  result  in  a  substantial  savings  in  time  when  considering  large 
networks. 

2.2.1  Summary  Table  1  is  a  summary  of  the  results  in  current  literature 
found  on  radar  emitter  classification  using  neural  networks.  In  the  table,  RBF  is 
short  for  Radial  Basis  Function,  and  MLP  is  short  for  Multi-Layer  Perceptron. 

2.3  Neural  Network  Hardware 

2.3.1  Introduction  Hardware  implementations  of  Artificial  Neural  Networks 
(ANNs)  has  been  a  growing  field  for  the  last  couple  of  years.  Several  hardware 
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models  of  ANN  implementations,  also  referred  to  as  neural  computers,  now  exist  for 
a  number  of  different  training  algorithms  and  network  types.  This  section  presents 
a  brief  survey  of  neural  network  hardware  found  in  current  literature. 

2.3.2  Survey  of  Current  Neural  Computing  Hardware 

2.3.2. 1  DTCNN  Chip:  Harrer  and  others  have  demonstrated  an  effi¬ 
cient  architecture  for  an  analog  realization  of  Discrete-Time  Cellular  Neural  Net¬ 
works  (DTCNN),  which  take  advantage  of  binary  outputs  and  simple  interconnec¬ 
tions  of  several  chips.  Because  this  chip  was  only  recently  developed,  no  test  per¬ 
formance  data  of  actual  realizations  have  been  reported  using  this  chip;  however, 
the  chip  is  being  considered  for  applications  such  as  discrete  convolution,  connected 
component  detection,  concentric  contouring,  and  oscillation  (10;466). 

2. 3. 2. 2  Neurocomputer  Chip:  Lont  and  Guggenbuhl  built  the  Neuro¬ 
computer  chip  that  contains  18  neurons  and  161  synapses  in  three  layers  and  pro¬ 
vides  16  inputs  and  4  outputs  (18:457).  The  complete  neural  computing  system 
consists  of  a  personal  computer  for  training  the  network  off-chip,  an  interface  card 
for  downloading  weights  to  the  chip,  and  a  Neurochip  (18:463). 

Experiments  with  the  Neurochips  have  been  limited  so  far;  however,  the  Neu¬ 
rocomputer  chip  has  been  tested  with  a  simple  pattern-recognition  application.  It 
was  found  that  at  best,  training  error  can  be  reduced  to  a  lineeir  error  of  11  percent. 
In  portability  tests,  the  authors  found  that  each  chip  has  to  be  retrained  individu¬ 
ally  because  no  two  chips  are  exactly  the  same  (18:463).  The  propagation  time  delay 
from  the  input  to  the  output  of  each  chip  is  equal  to  4  psec.  The  overall  computa¬ 
tion  rate  (weights  times  inputs)  is  40.25  million  connections  per  second.  Lont  and 
Guggenbuhl  state  that  this  computation  rate  is  not  the  effective  computation  rate 
because  the  weights  must  be  refreshed  from  time  to  time  (18:464). 
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2.3.2.3  ANNA  Chip:  Sackinger  and  others  designed,  fabricated,  and 
tested  a  reconfigurable  ANN  chip,  called  ANNA  (for  Analog  Neural  Network  Arith¬ 
metic  and  logic  unit)  (31:498).  The  ANNA  chip  is  optimized  for  locally  connected, 
weight-sharing  networks  and  time-delay  neural  networks. 

The  ANNA  chip  implements  4096  synapses  with  6  bits  of  resolution  for  each 
state  (input/output  of  the  neurons).  All  input  and  output  of  the  chip  are  digital, 
whereas  internally  the  chip  uses  analog  computing.  The  advantages  of  using  the 
ANNA  chip  over  most  other  implementations  are  high  synaptic  density,  high  speed, 
low  power  requirements,  and  ease  of  interfacing  to  a  digital  system  (31:498). 

The  processing  speed  on  the  chip,  from  start  to  finish,  takes  about  200  nsec 
(four  clock  cycles).  The  ANNA  chip  is  configured  for  weight  vector  sizes  of  64,  128, 
and  256,  which  also  correspond  to  the  number  of  synapses  per  neuron  (31:498).  For 
the  maximum  configuration  size  of  256  synapses  per  neuron,  the  chip  can  evaluate  a 
maximum  of  10  billion  connections  per  second.  For  practical  applications,  however, 
the  speed  will  be  lower  because  some  applications  will  not  make  full  use  of  the  ANNA 
chips’s  parallelism  (31:499). 

The  locally  connected,  weight-sharing  neural  network  (the  neural  network  de¬ 
signed  on  the  ANNA  chip)  has  been  successfully  used  for  optical  character  recog¬ 
nition  of  both  digits  and  letters,  for  character  recognition  from  touch  screens,  for 
image  segmentation,  and  for  speech  recognition  (31:498). 

2.3.24  TInMANN  Chip:  Melton  and  others  have  designed  the  archi¬ 
tecture,  operation,  and  implementation  for  The  Integer  Markovian  Artificial  Neural 
Network  (TInMANN)  chip,  which  features  a  massively  parallel  all-digital,  stochastic 
architecture  (21:375).  The  TInMANN  uses  a  modified  Kohonen  algorithm  to  simplify 
the  computations  on  chip.  The  integer  Markovian  learning  algorithm  replaces  mul¬ 
tiplications  and  complex  function  evaluations  with  simple  additions,  comparisons, 
and  random  number  generation  (21:377). 


12 


The  current  implementation  model  uses  one  neuron  per  chip,  which  can  take 
vectors  up  to  six  dimensions  at  10-bits  precision.  This  model  is  capable  of  pro¬ 
cessing  up  to  195,000  three-dimensional  training  vectors  per  second  (21:382).  The 
TInMANN  was  designed  to  be  flexible  and  capable  of  being  expanded  to  larger 
systems.  The  primary  drawbacks  with  the  TInMANN  chip  are  the  low  level  of  in¬ 
tegration  (only  one  neuron  per  chip),  the  higher  power  and  reliability  problems  as 
compared  to  other  Very- Large- Scale-Integration  (VLSI)  chips,  and  the  slow  speed  of 
the  network  (21:383). 

2.3. 2.5  ETANN  Chip:  Intel  has  developed  an  Electronically  IVainable 
Analog  Neural  Network  (ETANN),  the  80170NX  chip.  The  80170NX  chip  is  an 
electronic  device  that  performs  ANN  functions  in  hardware,  and  is  currently  the 
most  capable  ANN  device  developed  so  far. 

The  ETANN  chip  is  a  64-neuron,  10,240-synapse  electronically  trainable  paral¬ 
lel  data  processor,  known  as  an  analog  neural  network.  It  is  configured  for  mapping 
128  inputs,  of  which  64  are  feedback  inputs,  into  64  neuron  outputs.  The  ETANN 
chip  features  a  low  power  requirement  (5- volts  operation),  a  resolution  of  7  to  8  bits, 
a  dense  208-pin  PGA  (Pin  Grid  Array)  package,  and  a  non-volatile  weight  storage 
capability  (minimum  data  retention  lifetime  of  10  years)  (13). 

A  single  ETANN  chip  can  perform  more  than  2  billion  multiply-accumulate 
operations  (connections)  per  second.  Moreover,  processing  throughput  takes  about 
6  ps&c  (or  3  ys&c  per  layer),  near  real-time  for  pattern  recognition  applications.  By 
interconnecting  eight  chips,  systems  can  achieve  more  than  16  billion  connections 
per  second,  a  performance  level  that  exceeds  most  supercomputers.  Hence,  this  ca¬ 
pability  makes  the  ETANN  chip  a  prime  candidate  for  real-time  pattern  recognition 
and  signal  processing  applications  (13). 

Mumford,  Andes,  and  Kem  have  designed  a  neural  network  processing  system 
that  incorporates  an  ANN  as  a  subsystem  in  a  layered  hierarchical  architecture 
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(22:423).  The  ANN  hardware  used  in  the  system  design  is  Intel’s  ETANN  chip. 
The  Mod  2  Neurocomputer,  which  is  being  built  at  the  Naval  Weapons  Center,  is 
designed  to  be  a  neural  network  processing  system,  using  separate  individual  neural 
networks  as  subsystems  in  a  flexible,  modular  architecture  (22:423). 

The  Mod  2  Neurocomputer  is  a  reconfigurable,  general-purpose  neural  pro¬ 
cessor  that  can  be  easily  tested  in  an  interactive  manner  and  integrated  with  an 
imaging  sensor  for  real-time  integrated  processing  demonstrations  (22:424).  The  de¬ 
signers  have  not  completed  hardware  construction  at  this  time.  When  completed, 
the  Mod  2  Neurocomputer  will  be  a  demonstration  of  ANN  hardware  integration  into 
a  system  designed  to  support  parallel  processing  of  image  data  at  real-time  rates. 

2.4  Conclusion 

This  chapter  reviewed  literature  on  the  topics  of  radar  emitter  classiflration 
and  neural  network  hardware.  The  best  radar  emitter  classification  accuracy  for 
a  large-class  problem  was  achieved  by  Capt  Cameron  at  AFIT.  He  achieved  a  96 
percent  classification  accuracy  on  a  simulator  using  the  same  data  that  is  to  be  used 
in  this  research. 

A  survey  of  neural  network  hardware  covered  five  newly  developed  neural  com¬ 
puter  devices:  the  DTCNN  chip,  the  Neurocomputer  chip,  the  ANNA  chip,  the 
TInMANN  chip,  and  the  ETANN  chip.  This  survey  concluded  with  an  example  of 
a  system  integration  using  Intel’s  ETANN  chip.  Some  of  the  chips  are  still  in  the 
construction  phase  and  have  not  been  tested,  whereas  others  have  undergone  limited 
testing.  Besides  being  the  oldest  device  surveyed,  the  ETANN  chip  remains  as  one 
of  the  most  capable  chips  in  use  today. 
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III.  METHODOLOGY 


3.1  Introduction 

This  chapter  is  divided  into  three  sections;  ETANN  description  and  theory 
of  operation,  phase  one  research,  and  phase  two  research.  Each  phase  of  research 
covers  tests  and  analysis  that  require  development  of  software  tools  and  procedures 
for  implementing  the  ETANN  device  for  radar  emitter  classification. 

Phase  one  research  tests  require  performance  evaluations  of  computer  simu¬ 
lations,  hardware  characterizations,  and  network  analysis  of  a  16-feature,  3-class 
radar  emitter  identification  problem.  Phase  two  research  requires  solving  several  16- 
feature,  multi-class  radar  emitter  identification  problems  in  hardware — some  of  the 
same  problems  Capt  Cameron  solved  in  simulation  (5:18).  In  phase  one  research,  two 
computer  simulators  are  used  for  off-chip  training;  they  are  briefly  discussed  below. 
Phase  two  research  uses  only  one  simulator  for  the  final  hardware  implementation. 

3.1.1  Software  Simulators  The  two  simulators  used  in  this  resezirch  are  Neu¬ 
ral  Graphics  (37)(38)  and  iBrainMaker  (3).  Each  simulator  is  used  for  a  comparison 
of  speed  and  accuracy. 

The  Neural  Graphics  simulator  was  developed  by  Greg  Tarr  and  was  used 
in  his  doctoral  dissertation  for  research  and  modelling  of  neural  networks.  Neural 
Graphics  implements  a  neural  network  model  by  using  a  data  file  containing  both 
training  and  test  vectors,  also  known  as  training  and  test  patterns.  Neural  Graphics 
features  several  paradigms  for  training  various  kinds  of  networks.  Its  basic  back 
propagation  paradigm  was  modified  to  emulate  the  ETANN  transfer  function  and  is 
used  in  this  research  (see  Appendix  F). 

The  Neural  Graphics  simulator  provides  several  measures  of  network  accuracy 
while  performing  the  classification  task.  Most  importantly,  it  measures  a  percentage 
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right  and  good  for  both  training  and  test  data,  which  are  updated  according  to  the 
number  of  iterations  set  in  the  screen  update.  It  is  advised  that  the  updates  be 
set  to  reflect  training  epochs,  where  one  epoch  is  equal  to  the  number  of  tredning 
vectors  in  the  data  file  that  contains  both  training  and  testing  vectors.  The  term 
right  classification  refers  to  one  in  which  the  correct  output  node  value  is  greater 
than  0.9  and  all  other  nodes  have  values  less  than  -0.9.  The  less  strict  good  measure 
of  classification  is  defined  as  the  result  from  a  maximum  pick  and  is  used  throughout 
this  thesis,  termed  the  percentage  correct  Note:  These  values  were  later  changed  to 
0.89  and  -0.94  for  emulation  of  the  ETANN  device,  enhancement  of  training,  and 
reduction  of  total  error  (more  about  this  in  Chapter  4). 

The  BrainMaker  simulator  (v2.02),  which  was  developed  by  California  Sci¬ 
entific,  is  the  only  commercial  simulator  used  in  this  research.  The  iBrainMaker 
is  the  Intel  version  of  BrainMaker  adapted  for  interfacing  with  the  ETANN  chip 
programming  software,  known  as  Intel’s  Neural  Network  Training  System  (iNNTS). 

3.2  ETANN  Chip  Description 

Intel  developed  an  Electronically  Trainable  Analog  Neural  Network  (ETANN), 
the  80170NX  chip.  The  ETANN  chip  is  a  neural  computing  device  that  performs 
ANN  functions  in  hardware  and  is  the  primary  device  used  in  this  research.  A  brief 
description  of  the  ETANN  follows  below. 

The  ETANN  chip  is  a  64-neuron,  10,240-synapse  electronically  trcinable  paral¬ 
lel  data  processor,  known  as  an  analog  neural  network  or  analog  neural  computer.  It 
is  configured  for  mapping  128  inputs,  of  which  64  are  feedback  inputs,  into  64  neuron 
outputs.  The  ETANN  chip  features  a  low  power  requirement  (5- volts  operation),  a 
weight  resolution  of  7.5  bits  (20)(about  three  decimal  places  of  accuracy),  a  process¬ 
ing  resolution  of  6.5  bits  (20)  (between  two  and  three  decimal  places  of  accuracy), 
a  dense  208-pin  PGA  (Pin  Grid  Array)  package,  and  a  non-volatile  weight  storage 
capability  (minimum  data  retention  lifetime  of  10  years)  (13). 
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A  single  ETANN  chip  can  perform  more  than  2  billion  multiply-accumulate 
operations  (connections)  per  second.  Moreover,  processing  throughput  takes  about 
6  //sec  (or  3  /isec  per  layer)  for  a  two  layer  network,  which  is  near  real-time  for  many 
pattern  recognition  applications.  By  interconnecting  eight  chips,  systems  can  achieve 
more  than  16  billion  connections  per  second,  a  performance  level  that  exceeds  most 
supercomputers.  Hence,  this  capability  makes  the  ETANN  chip  a  prime  candidate 
for  real-time  pattern  recognition  and  signal  processing  applications  (13). 

3.2.1  ETANN  Theory  of  Operation  The  ETANN  chip  was  designed  so  that 
most  “learning”  is  performed  off  chip.  OfF-chip  learning  maximizes  flexibility  as  well 
as  speed  and  enhances  the  chip’s  life  by  reducing  on-chip  learning  time  (3:19-8). 
Chip-In- Loop  (CIL)  training  is  performed  on  the  chip  after  the  network  has  already 
been  trained  by  the  simulator  to  peak  performance.  CIL  is  used  to  compensate  for 
chip  variations  and  loss  of  resolution  when  going  from  simulator  to  chip.  This  process 
of  CIL  training  is  the  last  step  before  completing  the  training  process. 

The  ETANN  chip’s  normal  operation  can  be  described  as  Parallel  Distributed 
Processing  (PDP),  whereby  each  of  the  64  neurons  implements  the  following  function 
(13:7): 


Output(Ui)  =  Sigmoid{^  Weight{wij)Input{xj)  -f-  BIASik} 

j  k 

2 

2^  ^  ^'^{•ynaptie contributions)  ^ 

where  i  equals  the  neuron  number,  j  equals  the  number  of  inputs  to  neuron  i,  and  k 
equals  the  number  of  biases  to  neuron  i.  Although  the  ETANN  chip  has  16  bias  units 
per  neuron,  only  biases  0  through  6  are  used  in  normal  chip  operation.  Bias  units 
10  through  15  are  used  by  the  iNNTS  software  for  zeroing  the  sigmoidal  response  of 
neurons. 
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Figure  3.  ETANN  Block  Diagram  (13:2) 


The  equation  above  represents  the  inner  product  operation  that  is  “squashed” 
by  a  sigmoidal  transfer  function.  Figure  3  illustrates  a  block  diagram  of  the  ETANN 
chip.  There  are  32  fixed-input  bias  weights  and  128  vEiriable  input  weights  that 
comprise  160  synapses  per  neuron.  Each  neuron  produces  a  single  scalar  output 
(13:8). 

The  ETANN  chip’s  processing  resolution  of  the  analog  inputs  and  outputs  is 
greater  than  or  equal  to  6  bits  (about  two  to  three  decimal  places  of  accuracy), 
lypically  the  weight  resolution  is  at  about  7.5  bits  (or  about  three  decimal  places  of 
accuracy).  At  worst  over  a  lifetime  of  10  years,  the  processing  resolution  would  be 
greater  than  or  equal  to  4  bits.  If  the  bake/re-training  method  is  used  as  described 
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in  (13),  the  worst  processing  resolution  over  a  lifetime  of  10  years  would  be  greater 
than  6  bits.  The  bake/re-training  method  process  involves  baking  a  chip  in  an  oven 
heated  to  250*0  for  24  hours  and  then  re-training  it.  According  to  Intel,  the  bake/re- 
training  method  significantly  improves  the  ETANN  weight  retention  over  extended 
periods  of  time. 

The  weights  on  the  ETANN  chip  are  stored  as  analog  transconductance  values. 
Each  synapse  produces  an  analog  output  current  from  an  analog  input  voltage  and 
a  stored  weight  voltage.  The  currents  generated  by  each  of  160  synapses  along  a 
“dendrite”  are  summed  to  form  an  input  to  a  neuron.  This  sum  of  currents  is  then 
converted  to  a  voltage  and  used  to  activate  a  sigmoid  function,  which  has  a  voltage- 
controlled  gain.  When  in  the  128-input  mode,  the  input  and  feedback  arrays  are 
active  and  the  output  corresponds  to  the  sum  of  their  two  dot  products. 

Figure  4  illustrates  the  synapse  operation  on  the  ETANN  chip.  Each  synapse 
contains  a  multiplier  and  a  stored  weight,  which  are  used  to  calculate  one  product 
with  an  input.  All  the  single  products  are  summed  from  other  synapses  to  produce 
the  dot  product  (13:9). 

3.2. 1. 1  ETANN  Chip  PrxKessing  Configurations  Figure  5  illustrates  sin¬ 
gle  ETANN  chip  processing  and  shows  two  basic  configurations:  (a)  a  configuration 
where  64-dimensional  input  vectors  are  directly  mapped  into  64-dimensional  output 
vectors,  and  (b)  a  configuration  where  128-inputs  are  mapped  into  64  outputs  by 
multiplexing  the  analog  input  pins  Iq-Igs  to  the  feedback  array.  Pin  A12  determines 
the  mode  that  the  ETANN  chip  is  in,  either  multiplex  mode  or  direct  mode. 

Figure  6  illustrates  two  more  chip  configurations:  (a)  a  two-layer  processor, 
and  (b)  a  Hopfield  network.  These  configurations  are  also  possible  with  a  single 
ETANN  chip.  Several  other  configurations  are  published  in  (13). 


19 


SnUPK  MnKSSWTAnON 


matneuaucalhooo. 


■VNAMCaCMEIMTie 


Figure  4.  ETANN  Synapse  Implementation  (13:9) 
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Figure  5.  ETANN  Processing  Configurations  (13:11) 
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Figure  6.  ETANN  Processing  Configurations  (13:12) 
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S.2.1.2  ETANN  Mxdti-Chip  Processing  Configurations  Intel  provides 
two  basic  multi-chip  processing  configurations:  1)  Direct  Pin  Interconnect,  or  2)  Bus 
Interconnect.  The  more  strmghtforward  method  is  direct  pin-to-pin  because  no  mul¬ 
tiplexing  is  required.  Figure  7  illustrates  both  multi-chip  configurations.  Although 
direct  pin-to-pin  is  simpler,  bus  interconnection  provides  more  fiexibility.  Bus  inter¬ 
connection,  however,  has  two  drawbacks:  1)  data  can  be  moved  only  in  64- neuron 
blocks,  and  2)  multiplexing  is  slower  than  direct  pin-to-pin  due  to  clocking. 

3.2.2  ETANN  Transfer  Function  Characterization  Intel  published  the  fol¬ 
lowing  sigmoidal  transfer  function  characteristics  that  simulate  the  ETANN  chip  at 
what  Intel  termed  as  typical  operating  conditions  (for  binary  outputs):  Vgain  =  5V, 
yREFo  —  VhEFi  =  1.4V,  for  64  input  operation  (13:27).  Vgain  sets  the  gain  of  the 
device  sigmoid;  it  can  essentially  change  the  sigmoid  from  a  linear  function  over  the 
input  range  to  a  step  function  at  the  high  end  (4:9).  Vrefo  and  VnsFi  define  the 
null  voltage  levels  independently  for  inputs  and  outputs. 


Neuron  Output  =  - - 1 

H- e“® 


(2) 


Intel’s  more  accurate  model  (for  binary  outputs)  that  includes  roll-off  due  to 
large  input  and  weight  magnitudes  as  well  as  output  range  limitations  is  shown  below 
(13:27). 


NeurmOutpvt  =  (i'— W 

where  x,-  are  the  input  values  and  to,-  are  the  weight  values.  Inputs  are  constrained 
between  —1  <  Xi  <  1.  Weights  are  constrained  between  —2.5  <  to,  <  2.5.  For 
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Figure  7.  ETANN  Multi-Chip  Processing  Configuration  (13:15) 
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Figure  8. 


Neuron  Output  Transfer  Chararteristics  Versus  Summing  Current  and 
Vref  (13:25) 


non-binary  output  operation,  Intel  has  not  derived  an  accurate  sigmoidal  transfer 
function,  which  is  the  impetus  for  characterization  experiments. 

Figure  8  illustrates  the  neuron  output  transfer  characteristics  versus  the  sum¬ 
ming  current  and  Vgain  Vrefo  =  1.4V.  Each  of  the  curves  represents  the  neuron 
output  given  a  particular  V^a/at  value. 

The  Neural  Graphics  simulator  was  configured  with  the  transfer  function  in 
Equation  2,  but  the  network  failed  to  train  properly.  When  the  weights  were  exported 
from  Neural  Graphics  and  downloaded  on  the  ETANN  chip,  the  result  was  poor 
classification.  The  cause  of  this  discrepancy  was  not  clear  until  the  discovery  that 
the  VcAiff  for  the  published  transfer  function  was  set  at  S.OV  (used  for  binary  output 
operation  only),  whereas  the  Vgain  for  normal  sigmoid  operation  on  the  ETANN 
is  set  to  S.SOV.  Consequently,  the  operating  sigmoidal  transfer  function  was  not 
accurately  known  for  a  Vgain  =  3.30V. 
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According  to  (3),  iBrainMaker  uses  a  lookup  table  as  opposed  to  a  pure  math¬ 
ematical  transfer  function  to  emulate  the  ETANN  sigmoid.  A  transfer  function  was 
extrapolated  by  the  researcher  from  the  menu  information  given  in  the  program. 
Given  in  the  menu  are  the  high,  low,  and  gain  values  for  the  sigmoidal  function. 
Using  this  information,  the  look-up  table  (resolution  unknown)  probably  emulates 
the  following  sigmoidal  transfer  function: 


1  765 

iBrainMaker  Neuron  Output  =  - — - - -  ..  — r  —  .915  (4) 

*  j.  1.0  >  ^(jynflp<tc  confrtptihonj)  '  ' 

Note:  The  iBrainMaker  sigmoidal  transfer  function  was  extrapolated  from  the  high, 
low,  and  gain  values  after  performing  independent  characterization  experiments  in 
this  research  (see  Chapter  4  results). 

Researchers  who  desire  to  interface  their  own  custom  simulator  with  the  ETANN 
hardware  are  encouraged  to  characterize  the  ETANN  sigmoidal  transfer  function 
through  experimentation.  Chapter  4  presents  a  method  used  in  this  research  to 
characterize  the  ETANN  sigmoidal  transfer  function  along  with  findings  from  this 
research.  It’s  worth  mentioning  here  that  chip-in-loop  training  if  used  excessively, 
decreases  the  life  of  the  chip  (3);  therefore,  application  designers  should  optimize 
training  off-chip  so  as  to  minimize  on-chip  training. 

3.3  Phase  One  Research 

The  purpose  of  phase  one  research  was  to  get  software  and  hardware  tools 
working  to  solve  a  small  network  problem,  and  then  use  the  knowledge  learned  from 
phase  one  to  solve  more  complicated  problems  in  phase  two.  Phase  one  research  was 
essentially  a  developmental  step  to  solving  the  thesis  problem  as  outlined  in  Chapter 
1. 
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The  test  network  constructed  for  hardware  implementation  consisted  of  16 
features  and  3  classes.  The  number  of  feature  vectors  used  in  phase  one  research 
was  600.  This  number  ensured  that  Foley’s  rule  was  not  violated.  Foley’s  rule  states 
that  the  number  of  training  vectors  per  class  should  be  greater  than  three  times  the 
number  of  features  (27:61).  When  this  criteria  is  met,  the  observed  error  rate  on 
the  training  data  is  a  good  predictor  of  the  network  error  rate  on  an  independent 
set  of  test  data.  As  long  as  the  total  number  of  vectors  per  class  is  greater  than  48 
for  a  given  16-feature  training  set,  Foley’s  rule  will  not  be  violated.  From  the  total 
training  set  of  600  vectors  for  this  test  problem,  each  class  had  200  vectors,  which 
easily  satisfied  Foley’s  rule. 

3.3.1  Equipment  Required  The  required  equipment  for  this  research  included 
a  Silicon  Graphics  workstation  for  running  the  Neural  Graphics  software  simula¬ 
tions  and  Intel’s  Neural  Network  Training  System  (iNNTS)  for  interfacing  with  the 
ETANN  chip.  The  iNNTS  includes  an  Intel  486  personal  computer,  a  Generic  Pro¬ 
grammer  Interface  (GUPI),  a  Personal  Computer  Personal  Programmer  (PCPP) 
board  which  links  the  PC  to  the  GUPI,  an  80170NX  adapter  that  connects  the 
ETANN  chip  to  the  GUPI,  and  a  software  simulation  program  to  interface  with  iN¬ 
NTS  (iBralnMaker).  The  data  for  training  and  testing  the  ANN  was  provided  by 
the  Georgia  Tech  Research  Institute  (GTRI),  Atlanta,  Georgia. 

3.3.2  Software  Development  GTRI  supplied  the  data  in  an  ASCII  code  for¬ 
mat  for  this  research;  however,  the  file  format  was  incompatible  with  the  simulator 
formats  used  in  this  research.  Consequently,  the  GTRI  data  had  to  be  reformatted 
for  each  simulator,  shuffied  in  a  file,  and  statistically  normalized. 

This  task  was  broken  down  into  three  steps: 
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1.  Transform  the  GTRI  data  into  Neural  Graphics  and  iBrainMaker  formats. 
Two  C  programs  were  written  to  accomplish  this  first  step  and  are  included  in 
Appendix  D:  “gtri_to_ng.c”  and  “gtri-to-bm-c” . 

2.  Shuffle  the  lines  of  vectors  for  both  Neural  Graphics  and  iBrainMaker  formats. 
Two  C  programs  were  written  to  accomplish  this  second  step  and  are  also 
included  in  Appendix  D:  “shuffle.ng.c”  and  “shuffle.bm.c” . 

3.  Statistically  normalize  the  data.  The  type  of  normalization  used  was  a  normal 
Gaussian  distribution  that  was  “squashed”  (between  -1  and  1).  The  ETANN 
chip  must  be  trained  and  tested  by  data  dynamically  normalized  between  - 
1  and  1.  Two  C  programs  were  written  to  convert  the  data  to  two  simulator 
formats  discussed  previously  and  are  also  included  in  Appendix  E:  “norm.ng.c” 
and  “norm.bm.c”. 

3.3.3  Procedure  The  network  design  procedure  is  basically  a  seven-step  pro¬ 
cess  of  working  through  a  logical  description  of  the  problem  to  be  solved.  In  phase 
one,  the  problem  was  to  characterize  a  16-feature,  3-class  problem.  The  first  step 
in  the  design  process  is  to  determine  whether  the  network  is  to  predict,  generalize, 
or  recognize.  Prom  the  research  problem  presented  in  Chapter  1,  the  network  was 
supposed  to  recognize  radar  emitters.  The  second  step  requires  a  designer  to  choose 
information  that  will  make  good  features  for  recognition .  GTRI  performed  this  step 
and  the  following  step,  which  was  to  get  lots  of  data. 

Generally,  the  more  data,  the  better  the  predictor,  generalizer,  or  recognizer. 
In  some  cases,  too  much  data  will  overwhelm  the  network.  A  good  method  of 
determining  over  training  is  to  plot  the  total  error  for  the  network  versus  the  number 
of  training  iterations  or  epochs.  When  the  the  curve  begins  to  reverse  direction 
(higher  error),  it  is  time  to  stop  training. 
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The  next  step  is  to  build  the  network.  When  building  a  feed-forward  network, 
there  are  two  things  needed  to  make  the  network.  One  needs  samples  of  input  data 
(training  vectors)  and  to  know  the  desired  results. 

Two  things  are  needed  to  make  a  network:  a  network  definition  and  a  collection 
of  training  vectors  (34).  A  network  definition  consists  of  assigning  a  number  of  input 
nodes  to  a  number  of  hidden  nodes  and  a  number  of  output  nodes.  The  assigning  of 
input  nodes  and  output  nodes  is  problem  dependent.  The  assigning  of  hidden  nodes 
is  problem  and  data  dependent.  If  the  problem  is  linearly  separable,  then  hidden 
nodes  are  not  necessary.  The  goal  is  to  train  effectively  with  as  few  hidden  nodes  as 
possible,  so  as  to  minimize  the  number  of  computations  in  the  network. 

After  building  the  network  comes  training  the  network.  TVaining  a  network 
is  more  an  art  than  a  science.  Adjusting  learning  parameters  such  as  momentum, 
learning  rate,  and  tolerance  all  become  factors  in  how  fast  a  network  will  converge, 
if  it  ever  does. 

Testing  the  network  follows  training  the  network.  A  percentage  of  the  data 
is  always  saved  as  testing  data.  Generally,  10-20  percent  of  the  total  number  of 
vectors  is  sufficient  to  test  the  network.  It  is  important  to  show  the  network  data 
that  it  has  not  seen  before,  because  only  then  will  one  know  if  the  network  is  a  good 
predictor,  generalizer,  or  recognizer.  The  last  step  in  this  process  is  running  the 
network,  presenting  it  with  new  input  data,  and  gathering  usable  results.  At  this 
point,  one  has  completed  the  seven-step  design  process. 

3.4  Phase  Two  Research 

In  phase  two,  a  solution  to  the  main  problem  of  characterizing  30  radar  emitters 
in  hardware  was  the  focus  of  research  and  analysis.  Using  what  was  learned  in  phase 
one,  the  task  was  simplified  to  that  of  using  the  same  basic  tools  and  procedures. 
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3.4.1  Equipment  Required  The  required  equipment  for  phase  two  research 
was  the  same  as  that  described  in  phase  one  with  the  addition  of  using  the  ETANN 
Multi-Chip  Board  (EMB)  for  multi-chip  prototyping.  The  data  for  training  and 
testing  the  networks  in  phase  two  were  an  extension  of  what  was  used  in  phase  one, 
which  included  all  30  classes. 

3.4. S  Software  Development  Software  developed  in  phase  one  research  was 
extended  in  phase  two  for  construction  of  larger  networks.  The  programs  used  in 
phase  one  were  modified  to  work  in  phase  two.  The  main  difference  in  the  programs 
is  the  handling  of  more  classes  and  pattern  vectors. 

3.4.3  Procedure  The  procedures  for  designing  an  artificial  neural  network  are 
basically  the  same  as  those  discussed  in  phase  one,  but  the  problem  was  much  more 
complicated  by  the  fact  that  the  number  of  classes  was  increased.  Capt  Cameron 
published  a  baseline  of  test  findings  that  are  used  here  as  a  comparison  with  hardware 
implementation  results  (5). 

3.4.4  Baseline  Tests  Capt  Cameron  performed  a  series  of  experiments  with 
the  Neural  Graphics  simulator  to  provide  a  baseline  against  which  to  compare  ex¬ 
perimental  results.  Accordingly,  the  number  of  classes  per  data  file  and  the  number 
of  vectors  per  class  were  both  varied  as  widely  as  possible.  In  addition,  the  class 
numbers  were  reassigned  to  ensure  a  random  combination  of  classes.  The  transfer 
function  used  in  Cameron’s  simulation  was  a  sigmoidal  function  equal  to  l/(l-f  e~°) 
over  the  interval  [0,-f  1].  The  ETANN  chip  is  characterized  by  another  sigmoidal 
transfer  function,  which  will  be  discussed  in  Chapter  4.  The  baseline  of  experimen¬ 
tal  results  is  included  here  with  note  of  the  following  information: 

•  Classes;  randomly  selected,  numbers  as  indicated  in  Table  2 

•  Training  vectors:  as  indicated  in  table 

•  Test  vectors:  25  per  class 
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Table  2.  Baseline  Testing  (5:18) 


TVaining 

Percentage  Correct  (Test/Train) 

Vectors 

Iterations 

Number  of  classes 

(per  class) 

(x  1000) 

8 

14 

20 

26 

30 

WtSM 

96/98 

96/95 

84/88 

84/83 

97/100 

96/98 

88/91 

88/89 

87/89 

25 

hh 

98/100 

96/99 

91/94 

88/90 

WKSM 

96/99 

82/80 

96/99 

95/98 

89/92 

92/94 

90/92 

50 

91/96 

92/95 

89/92 

MSM 

96/97 

97/97 

86/88 

82/83 

insuf.  data 

97/99 

97/98 

89/92 

92/93 

insuf.  data 

75 

97/99 

98/98 

91/94 

92/93 

insuf.  data 

100 

WBM 

iiil 

96/97 

78/79 

insuf.  data 

100 

97/97 

92/93 

insuf.  data 

100 

hh 

iiSI 

97/98 

91/95 

92/93 

insuf.  data 

98/99 

96/96 

76/77 

insuf.  data 

Bl 

98/99 

96/98 

insuf.  data 

125 

97/98 

91/92 

insuf.  data 

•  Hidden  nodes:  16 

•  Iterations:  as  indicated 

•  Special  note:  some  classes  have  too  few  vectors;  thus  this  is  why  there  are 
insufficient  data  entries 


In  Table  2,  it  is  evident  that  classification  accuracy  increases  as  the  number 
of  training  vectors  increase,  and  the  accuracy  decreases  as  the  number  of  classes 
increase,  as  expected.  Accuracy  generally  increases  as  the  number  of  iterations 
increase,  but  the  effect  is  more  pronounced  for  runs  with  twenty  or  more  classes. 
This  result  is  expected  since  the  larger  networks  had  to  be  trained  with  more  vectors. 
The  maximum  baseline  accuracy  recorded  for  a  single  network,  30-class  problem 
is  90  percent.  Of  course,  this  result  is  not  Capt  Cameron’s  best  result.  Using  a 
hierarchical  method,  he  achieved  a  96  percent  accuracy  with  an  18(top)/12(bottom) 
split  of  classes  in  8,469  test  vectors  representing  half  of  each  class  (5). 
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3.5  Conclusion 


This  chapter  has  given  the  reader  a  description  of  the  ETANN  chip,  a  synopsis 
of  phase  one  and  two  research  objectives,  and  a  development  plan  for  software  tools 
and  procedures  needed  for  implementing  a  neural  network  device  to  solve  a  radar 
emitter  identification  problem.  In  addition,  a  baseline  of  experimental  results  from 
(5)  was  included  here  as  a  bench-mark  for  device  implementation. 
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IV.  RESULTS 


4.1  Introduction 

This  chapter  presents  the  experimental  results  for  the  problems  that  were  dis¬ 
cussed  in  Chapter  1.  Phase  one  research  findings  include  the  sigmoidal  transfer 
function  characterization  results,  simulator  test  comparisons,  and  hardware  imple¬ 
mentation  tests.  Phase  two  research  findings  include  single  chip  and  multi-chip 
implementations  using  Intel’s  ETANN  Multi-chip  Board  (EMB). 

4.2  Phase  One  Research  Findings 

4.2.1  ETANN  Sigmoidal  Transfer  Function  Characterization  Experiment  The 
general  procedures  for  the  ETANN  sigmoidal  transfer  function  characterization  ex¬ 
periment  are  outlined  as  follows: 

1.  Interface  with  Intel’s  Neural  Network  Training  System  to  perform  read  and 
write  functions  on  the  chip’s  inputs  and  synapses. 

2.  Prepare  the  ETANN  device  for  programming  by  using  the  “prepare  chip”  com¬ 
mand  in  iNNTS — use  the  following  settings:  Vqain  =  3.3V^  and  Vrefi  = 
Vrefo  =  l-SV,  64-input  operation,  number  of  initialization  biases  =  9,  and 
weight  precision  set  to  0.042. 

3.  Perform  a  series  of  tests  on  a  number  of  avzdlable  chips  (eight  were  used  for  this 
experiment)  over  a  range  up  to  the  maximum  input  values  (input  values  =  -1.0, 
-0.75,  -0.5,  -0.25,  0.0,  0.25,  0.5,  0.75,  1.0)  and  weight  values  (weights  values= 
-2.5,  -2.0,  -1.5,  -1.0,  -0.5,  0.0,  0.5,  1.0,  1.5,  2.0,  2.5)  in  order  to  calculate  the 
average  ETANN  chip  response. 

4.  Write  input  values  to  a  single  ETANN  neuron  with  varying  numbers  of  inputs 
(e.g.,  1,  2,  16,  and  32). 
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5.  Calculate  the  average  hardness  parameter  (a  sigmoid  gain  coefficient)  for  each 
experiment  of  varying  inputs  (hardness  parameter  equals  eight  in  Intel’s  char¬ 
acterization  equations). 

6.  Plot  activation  values  versus  output  values  to  characterize  the  sigmoidal  trans¬ 
fer  function  of  the  ETANN  chip. 

7.  Write  a  program  to  calculate  the  mean-square-error  (MSE)  versus  the  hardness 
parameter  changes,  and  plot  the  results  for  various  number  of  inputs  to  a  single 
neuron. 

8.  Develop  a  mathematical  model  for  the  transfer  function,  and  code  the  function 
in  Neural  Graphics  (or  your  favorite  simulator). 

9.  TVain  a  network  using  a  simulator  with  the  newly  characterized  sigmoidal  trans¬ 
fer  function,  and  then  download  the  weights  to  an  ETANN  device  for  on-chip 
testing. 

4.S.2  ETANN  Sigmoidal  Transfer  Function  Characterization  Findings  Pro¬ 
gramming  in  C  was  necessary  for  weight  format  conversion  between  Neural  Graphics 
and  Intel’s  Neural  Network  Training  System  (iNNTS).  In  addition,  several  C  pro¬ 
grams  were  written  to  analyze  and  process  data  collected  from  iNNTS;  a  sample  of 
these  programs  may  be  found  in  Appendix  G. 

Figure  9  illustrates  a  data  fit  to  the  sigmoidal  transfer  function,  which  was 
characterized  by  averaging  data  collected  from  eight  ETANN  chips.  In  each  of  the 
plots,  the  x-axis  is  called  the  activation,  and  the  y-axis  is  called  the  output.  The 
activation  represents  a  sigmoidal  function  acting  on  the  summation  of  all  inputs 
multiplied  by  associated  weights  plus  biases.  The  biases  in  this  experiment  were  set 
to  zero  during  the  prepare  chip  step.  The  inputs  were  written  to  a  single  neuron 
(all  tests  were  performed  on  a  single  neuron)  in  incremental  steps  (input  values  = 
-1.0,  -0.75,  -0.5,  -0.25,  0.0,  0.25,  0.5,  0.75,  1.0).  The  weights  were  changed  over  a 
range  of  11  values  from  maximum  negative  to  maximum  positive  values  (weights 
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values=  -2.5,  -2.0,  -1.5,  -1.0,  -0.5,  0.0,  0.5,  1.0,  1.5,  2.0,  2.5).  For  example,  the  first 
weight  value  chosen,  -2.5,  was  fixed  and  then  the  inputs  (depending  on  how  many 
were  selected  for  that  particular  test)  were  written  to  a  single  neuron,  incrementing 
through  the  entire  range  of  nine  input  values.  Then  the  weight  was  changed  (set  to 
-2.0,  the  next  value).  The  inputs  were  written  to  a  single  neuron  again,  incrementing 
through  the  entire  range  of  nine  values.  This  process  was  repeated  for  all  11  weight 
values. 

Additionally,  each  plot  in  Figure  9  shows  the  result  of  a  particular  test  (testl, 
test2,  tests,  and  test4  as  labeled  on  the  plot  headings).  Testl  represents  a  single 
input  to  neuron  zero  of  each  of  the  eight  ETANN  chips.  Test2  represents  two  inputs 
to  neuron  zero  of  each  of  the  eight  ETANN  chips.  Tests  and  test4  represent  16  and 
S2  inputs  respectively  to  neuron  zero  in  each  of  the  eight  ETANN  chips. 

For  each  test,  a  different  hardness  parameter  was  found  by  searching  for  the 
minimum  MSE  over  a  range  of  various  hardness  parameter  values  between  zero  and 
nine.  Table  S  shows  the  hardness  characterization  experiment  results  for  a  Vcain  — 
S.SV  and  a  Vrefo  =  VREFi  —  l-SV  (64-input  operation).  For  each  activation  value, 
a  numerical  output  was  automatically  measured  and  recorded  in  a  data  file  along 
with  the  summation  of  inputs  multiplied  by  the  weight  value  (sum  of  products), 
which  was  fixed  for  the  entire  range  of  inputs.  This  process  was  repeated  until  all  11 
weight  values  were  used.  The  plots  in  Figure  9  illustrate  activation  versus  output  (99 
points  in  tests  one  and  two  and  fewer  in  three  and  four  due  to  limiting  of  the  x-axis 
to  [-8:8]).  The  averaged  data  represents  the  average  activation  response  from  eight 
ETANN  chips  for  the  nine  input  values  and  11  weight  values.  Tests  three  and  four 
results  show  erratic  behavior  near  zero  activation  values  (see  plots  C  and  D).  The 
behavior  is  a  reaction  to  changing  weight  values  (over  the  1 1  values  mentioned  above) 
with  zero  (or  near  zero)  input  values.  This  erratic  behavior  might  be  explained  by 
noting  that  as  more  inputs  are  added  to  a  neuron,  a  larger  current  is  drawn  and  thus 
the  transfer  function  characteristics  change — modelling  such  change  is  difficult  if  not 
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Table  3.  Hardness  Characterization  Results 


Experiment 

Number  of 
Neuron  Inputs 

MSE 

Hardness 

Parameter 

1 

1 

0.00412 

1.7 

2 

2 

0.00355 

1.5 

3 

16 

0.00449 

1.2 

4 

32 

0.01235 

0.9 

impossible  for  all  network  variations;  therefore,  an  average  value  for  these  points  was 
taken  as  the  overall  slope  coordinate. 

In  Figure  10  the  hardness  parameter  versus  MSE  is  plotted  over  the  range 
[0.0:9.0]  for  all  tests.  In  training  experiments  using  the  four  different  hardness  pa¬ 
rameters  found  in  the  MSE  tests,  neural  networks  for  this  problem  trained  to  a 
lower  total  error  while  using  the  largest  hardness  parameter  (from  experiment  one) 
to  characterize  the  transfer  function.  Tests  two,  three,  and  four  show  that  the  trans¬ 
fer  function’s  output  slope  decreases  slightly  with  more  inputs,  but  the  summation 
of  synaptic  contributions  usually  exceeds  the  dynamic  range  where  the  hardness  pa¬ 
rameter  has  the  most  effect  on  the  output.  For  example,  if  a  network  has  16  inputs, 
the  dynamic  range  of  the  activation  function  parameter  a  (sigmoid [a])  is  between 
—40  <  a  <  40,  where  a  is  the  summation  of  weights  times  inputs  (assuming  biases 
are  set  to  zero  in  this  example).  The  sigmoidal  function’s  slope  region  is  most  pro¬ 
nounced  between  —  2  <  a  <  2,  which  is  a  very  small  region  compared  to  the  entire 
dynamic  range.  As  the  number  of  inputs  increase,  the  transfer  region  becomes  less 
significant  in  the  sigmoid  output.  Therefore,  using  the  leurgest  hardness  parameter 
measured  for  all  network  sizes  should  give  the  best  result  for  simulation  without 
having  to  adapt  the  hardness  parameter  for  different  sized  networks. 

4.2.2. 1  ETANN  Sigmoidal  Transfer  Function  Based  on  experimental 
results  for  a  Vqain  =  3.3V  and  a  Vrefo  =  Vrefi  =  1-5V  (64-input  operation),  the 
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C.  Curve  Fit  for  16  Inputs  D.  Curve  Fit  for  32  Inputs 


Figure  9.  Curve  Fit  for  Various  Inputs 
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A.  MSE  Plot  for  1  Input 


B.  MSE  Plot  for  2  Inputs 
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C.  MSE  Plot  for  16  Inputs 


D.  MSE  Plot  for  32  Inputs 
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following  transfer  function  suffices  to  characterize  the  ETANN  device  for  simulation. 


1  83 

Neuron  Output  =  ,  ^  -  -94  (5) 

Note  that  the  hardness  parameter  which  provided  the  best  results  (lowest  error  in 
training  the  test  network)  was  derived  from  test  number  one  for  a  single  neuron. 

4.2.3  Simulator  and  ETANN  Implementation  Results  for  a  3-Class  Problem 
Simulation  and  implementation  tests  were  performed  to  assess  the  accuracy  of  the 
ETANN  sigmoidal  transfer  function  characterized  above.  Tables  4  and  5  illustrate 
simulator  and  implementation  results  (averaged  over  three  runs)  while  using  iBrain- 
Maker  (v2.02)  and  Neural  Graphics  as  simulators.  The  Neural  Graphics  simulator 
was  coded  with  the  sigmoidal  transfer  function  found  in  this  research.  The  iBrain- 
Maker  simulator,  which  is  a  commercial  software  system  that  comes  with  iNNTS 
(3),  uses  a  look-up  table  rather  than  a  coded  sigmoid;  using  the  high,  low,  and  gain 
values  that  (3)  provided  in  their  transfer  function  menu,  the  look-up  table  (resolution 
unknown)  probably  emulates  the  following  transfer  function: 


1  765 

iBrainMaker  Neuron  Output  - — : - r—. — r  —  ,915  (6) 

^  j  ^  ^—\A}^($ynapt%ccontTthut%on»)  '  ' 

Note:  The  iDynaMind  (v2.0)  (which  is  also  a  part  of  iNNTS)  was  not  included 
in  the  test  because  for  large-class  problems  with  thousands  of  training  vectors  (to 
be  used  in  later  experiments),  the  simulator’s  limitations  are  exceeded. 

Comparative  testing  was  done  to  check  Neural  Graphics’  results  with  those  of 
a  working  simulator,  iBrainMaker.  Both  simulators  trained  on  the  same  data  and 
were  also  tested  with  the  same  test  data;  the  initial  weights  were  randomized  for 
both  simulators.  After  training,  the  weights  were  downloaded  from  each  simulator 
to  an  ETANN  device  (reused  the  same  device)  for  testing.  Again,  the  same  test  data 
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Table  4.  ETANN  and  iBrainMaker  with  3-Classes 


Number  of 
Epochs 

Training 

Time(min:sec) 

Simulator 

Training 

Simulator 

Test 

ETANN 

Test 

18 

82% 

75% 

37 

88% 

78% 

51 

89% 

92% 

83% 

was  used  on  the  device  in  each  independent  test  without  further  training  to  check 
classification  of  test  data.  Three  independent  tests  were  performed  to  establish  a 
classification  average.  The  Neural  Graphics  simulator  test  results  were  matched  by 
ETANN’s  test  result.  The  iBrainMaker  simulator  performance,  however,  was  less 
than  ideal.  The  ETANN  tests  using  iBrainMaker’s  trained  weights  were  on  average 
15  percent  lower  than  those  tests  using  Neural  Graphic’s  weights  for  the  3-class 
network.  The  set-up  configuration  for  Neural  Graphics  and  iBrainMaker  was  as 
follows: 

•  TVaining  vectors:  540  (class  1  =  180;  class  2  =  177;  class  3  =  183) 

•  Test  vectors:  60  (class  1  =  20;  class  2  =  23;  class  3  =  17) 

•  Features:  16  (Radar  Signal  Characteristics  (5)) 

•  Hidden  nodes:  20 

•  Classes:  3  (number  of  emitters) 

•  Iterations:  540  iterations  equals  one  epoch  in  this  network 

•  Note:  In  Table  4  the  time  listed  includes  only  training  time. 

•  Note:  In  Table  5  the  total  training  time  plus  testing  time  is  listed.  After  each 
epoch  of  training.  Neural  Graphics  automatically  performs  a  network  test  using 
all  test  vectors. 
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Table  5.  ETANN  and  Neural  Graphics  with  3-Classes 


Number  of 
Epochs 

Training 

Time(min:sec) 

Simulator 

Training 

Simulator 

Test 

ETANN 

Test 

18 

91% 

93% 

93% 

37 

95% 

93% 

93% 

51 

IBSSU 

97% 

95% 

95% 

Simulation  and  ETANN  Implementation  Results  for  an  8-Class  Problem 
Table  6  and  7  show  the  test  results  for  an  8-class  problem.  These  eight  classes  are 
a  subset  of  the  first  eight  classes  of  the  30-class  radar  emitter  data.  As  clearly 
indicated  from  the  data  in  Table  6,  iBrainMaker  had  a  difficult  time  training  on  the 
data.  There  is  no  known  explanation  as  to  why  iBrainMaker  performed  so  poorly;  on 
the  other  hand,  Neural  Graphics  did  a  superb  job  of  training  on  the  exact  training 
and  test  data  used  in  iBrainMaker. 

Table  8  shows  the  Chip-In-Loop  (CIL)  training  results  with  iBrainMaker  on 
the  8-class  problem.  CIL  training  results  were  averaged  over  three  separate  training 
events,  each  totaling  1,842  epochs  of  training.  The  highest  on-chip  test  result  was 
88  percent,  which  is  10  percent  lower  than  the  on-chip  test  result  after  training  with 
Neural  Graphics. 

Because  of  time  constraints,  it  was  decided  after  this  test  that  further  training 
and  testing  ofiF-chip  would  be  limited  to  the  Neural  Graphics  simulator  even  though 
there  is  no  CIL  capability  at  this  time.  The  iBrainMadcer  simulator  trains  much  too 
slowly  for  large  class  problems  with  a  lot  of  data.  In  addition,  it  produces  poor 
classification  results  from  the  radar  emitter  data  used  in  this  research,  even  though 
CIL  training  was  used  to  improve  on-chip  classification  performance. 
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Table  6.  ETANN  and  iBrainMaker  with  8-Classes 
400  TVaining  Vectors  and  200  Test  Vectors 


Number  of 
Epochs 

Training 

Time  (hr:min:8ec ) 

Network 

Size 

Simulator 

Training 

Simulator 

Teat 

ETANN 

Test 

1,843 

2:33:00 

16x16x8 

71% 

72% 

4,774 

6:37:57 

16x16x8 

72% 

72% 

2,045 

4:37:19 

16x20x8 

80% 

76% 

77% 

Table  7.  ETANN  and  Neural  Graphics  with  8-Classes 
400  TVaining  Vectors  and  200  Test  Vectors 


Number  of 
Epochs 

Training 

Time(min:8ec) 

Network 

Size 

Simulator 

Training 

Simulator 

Test 

ETANN 

Test 

100 

01:40 

16x16x8 

95% 

94% 

98% 

Table  8.  CIL  Results  with  iBrainMaker 
400  TVaining  Vectors  and  200  Test  Vectors 


Simulator 

Epochs 

Network 

Size 

Simulator 

Train/Test 

CIL 

Epochs 

CIL 

Train/Test 

1,842 

16x16x8 

73%/69% 

0 

70%/65% 

11 

V 

mm 

88%/84% 

11 

11 

mSm 

89%f79% 

« 

11 

11 

89%/83% 

17 

11 

W 

40 

90%/87% 

W 

11 

11 

50 

90%/85% 

« 

11 

11 

60 

92%/86% 

11 

11 

11 

70 

92%/86% 

11 

11 

11 

80 

94%/86% 

19 

11 

11 

90 

90%/88% 

11 

11 

11 

100 

88%/69% 
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Table  9.  Train’ng  and  Testing  Results  I, 


Iterations 

(xlOOO) 

Classes 

Training 
Vectors 
(  Class/Total) 

Test 

Vectors 

(Class/Total) 

Total 

Error 

Neural  Graphics 
Simulator 
(Train/Test) 

ETANN 
(Test  Only) 

70 

19 

94%/91% 

92% 

61 

24 

94%/94% 

92% 

67 

14 

30 

94%/93% 

95% 

84 

14 

34 

93%/93% 

93% 

74 

20 

25/500 

43 

91%/89% 

88% 

86 

20 

25/500 

52 

87%/85% 

86% 

78 

26 

25/650 

61 

76%/76% 

72% 

98 

26 

25/650 

48 

82%/81% 

59% 

100 

30 

25/750 

45 

81%/80% 

72% 

100 

30 

25/750 

56 

78%/78% 

77% 

4.3  Phase  Two  Research  Findings 

4.3.1  Single  Chip  Tests  Table  9  is  a  summary  of  first-cut  results  while  using 
Neural  Graphics  to  simulate  the  ETANN  device  for  a  real  application — solving  the 
radar  emitter  identification  problem.  Note:  All  test  results  were  averaged  over  three 
independent  training/test  runs.  The  single  chip  tests  (as  well  as  multi-chip  tests) 
involved  training  using  the  normal  back  propagation  algorithm  (no  adaptive  mea¬ 
sures  or  momentum).  The  best  training  result  between  0  and  100,000  iterations  was 
chosen — that  is  the  reason  for  the  difierence  in  iteration  numbers  in  Table  9.  The 
number  of  training  and  test  vectors  were  chosen  to  correspond  to  the  baseline  results 
listed  in  Chapter  3.  As  the  number  of  classes  increase,  the  ETANN  test  results  get 
worse  (in  general).  The  chip  resolution  is  limited  at  about  three  decimal  places  of 
accuracy,  whereas  the  simulator  trains/tests  with  approximately  six  decimal  places 
of  accuracy.  Although  the  results  so  far  do  not  come  close  to  matching  baseline 
results  listed  in  Chapter  3,  these  results  are  a  starting  point  to  improve  upon. 

Several  training  improvement  techniques  were  tried  in  order  to  get  the  best 
training  and  testing  results.  First  of  all,  the  weights  in  the  Neural  Graphics  simu- 
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lator  were  growing  steadily  as  training  iterations  increased.  The  ETANN  device  is 
constrained  to  accept  weight  values  between  —2.5  <  Wij  <  2.5.  The  first  approach 
to  solve  the  problem  was  to  scale  the  weights  after  training  and  write  them  to  the 
ETANN  device.  An  experiment  involving  real  training  data  was  performed  as  an  at¬ 
tempt  to  solve  the  problem.  Failure  was  the  result.  Apparently  a  lot  of  information 
is  lost  when  scaling  the  weights;  thus,  poor  testing  performance  can  be  expected. 

The  next  test  involved  changing  the  Neural  Graphics  weight  update  code.  In 
order  to  keep  the  weights  within  range  of  the  ETANN  device,  the  Neural  Graphics 
weight  update  program  routine  was  modified  to  limit  the  weight  values.  The  mod¬ 
ified  program  routine  is  in  Appendix  F.  The  biases,  on  the  other  hand,  are  aillowed 
to  exceed  the  range,  —2.5  <  Wij  <  2.5,  because  the  ETANN  device  allows  the  pro¬ 
grammer  to  use  up  to  seven  biases  per  neuron.  Although  Neural  Graphics  trains 
with  one  bias  per  neuron,  the  bias  value  may  be  divided  by  any  integer  between  smd 
including  one  and  seven.  For  this  research,  six  was  the  deviser  used,  and  the  result 
was  written  to  six  biases.  Consequently,  the  bias  values  in  Neural  Graphics  were 
limited  to  a  value  of  15. 

To  improve  network  performance  and  lower  the  total  error,  the  Neural  Graph¬ 
ics  code  was  modified.  Recall  the  ETANN  sigmoidal  transfer  function  experiment 
results.  Based  on  test  results,  it  was  concluded  that  the  ETANN  device  incurs  more 
error  when  training  to  -1-1  for  the  desired  class  and  to  —1  for  the  non-desired  classes; 
consequently,  if  the  actual  device  high  and  low  threshold  values  were  assigned  as  the 
desired  class  and  non-desired  classes,  respectively,  then  total  error  could  be  reduced 
in  training.  In  simulator  tests  performed  for  this  research,  training  results  did  im¬ 
prove  and  the  total  error  dropped  significantly  when  training  to  the  modified  desired 
and  non-desired  levels  (compare  training  zmd  testing  results  I  and  II).  The  ETANN 
levels  were  characterized  earlier  in  this  chapter  as  0.89  (high)  and  -0.94  (low)  and 
were  coded  in  the  Neural  Graphics  simulator  for  training/testing  improvement. 
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Further  improvements  in  training  were  made  by  using  the  exact  derivative 
calculated  from  the  sigmoidal  transfer  function,  denoted  by  /,  derived  from  the 
ETANN.  The  derivative  equation  is  the  following: 

df  {1.74a)  a{-1.83(l  +  -  .94} 

da  ~  da 

=  .95[.94  +  /(1.74a)][.89- /(1.74a)]  (8) 

The  result  of  using  this  derivative  function  as  opposed  to  the  precise  derivative 
of  a  tanh  function  (see  Appendix  A)  was  increased  training  speed  and  accuracy 
while  using  Neural  Graphics — see  Table  10.  Although  not  perfect  results,  this  table 
shows  a  definite  improvement  over  Table  9.  As  the  number  of  classes  increased 
in  the  data  sets,  classification  accuracy  decreased  as  expected  with  few  exceptions. 
Generally,  as  the  number  of  training  vectors  are  increased,  the  number  of  required 
training  epochs  increases;  this  is  why  there  were  some  exceptions  to  increasing  classes 
and  decreasing  classification  accuracy.  The  ETANN  classification  results  generally 
followed  the  same  trend  as  the  simulator’s,  except  for  the  two  30-class  networks, 
which  were  the  trend  breakers.  Resolution  breakdown  is  a  probable  explanation  for 
this  experiment  outcome.  The  ETANN’s  resolution  is  about  7.5  bits  for  the  weight 
values,  whereas  the  Neural  Graphics  simulator  has  more  than  32  bits  resolution. 
There  will  be  a  margin  of  error  regardless  of  what  simulator  a  person  uses,  but  the 
goal  is  to  minimize  that  error,  which  is  one  of  the  main  objectives  of  this  research. 
Fine  tuning  can  always  be  done  through  chip-in-loop  training,  which  is  the  final 
step  towards  realizing  a  fully  trained  network.  Chip  variation  and  resolution  can 
iisually  be  compensated  for  by  the  chip-in-loop  training  method,  but  poor  training 
and  testing  results  cannot. 

According  to  (28),  if  enough  hidden  neurons  are  used  in  a  two  layer  network 
(input X  hidden  x  output),  most  networks  can  be  accurately  classified  (given  good 
features  and  separable  data).  An  experiment  was  performed  to  find  an  efficient  num- 
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Table  10.  'IVaining  and  Testing  Results  II. 


Iterations 

(xlOOO) 

Classes 

Training 
Vectors 
(  Class/Total) 

Test 

Vectors 

(Class/Total) 

Total 

Error 

Neural  Graphics 
Simulator 
(Train/Test) 

ETANN 
(Test  Only) 

40 

25/200 

11 

95%/94% 

98% 

40 

25/200 

15 

94%/96% 

97% 

40 

14 

25/350 

15 

96%/94% 

94% 

40 

14 

25/350 

20 

95%/95% 

96% 

40 

20 

25/500 

20 

95%/93% 

91% 

40 

20 

25/500 

29 

92%/92% 

92% 

40 

26 

25/650 

29 

90%/89% 

92% 

40 

26 

25/650 

37 

90%/89% 

65% 

40 

30 

25/750 

38 

88%/88% 

71% 

40 

30 

25/750 

48 

88%/89% 

80% 

ber  of  hidden  neurons  that  would  give  good  classification  results  for  the  radar  emitter 
features.  The  data  was  trained  using  the  standard  back  propagation  algorithm.  Ta¬ 
ble  11  shows  that  generally  as  the  number  of  hidden  neurons  is  increased  the  total 
error  goes  down,  and  the  classification  accuracy  goes  up.  The  principal  drawback  to 
adding  more  hidden  neurons  is  slower  training  speed.  By  observation,  the  ETANN 
chip  appears  to  have  some  difficulty  in  achieving  better  than  83  percent  classification 
for  the  30-class  network.  A  difference  in  chip  resolution  versus  simulator  resolution 
is  a  likely  reason  for  the  discrepancy.  In  experiments  with  fewer  than  26  classes, 
the  ETANN  performed  well  for  this  set  of  data.  Classification  results  will  always 
be  problem  dependent — the  number  of  training  and  test  vectors,  the  separability  of 
classes,  and  the  quality  of  features  used  are  all  important  considerations. 

Table  12  shows  the  fin£d  training  and  testing  results  for  a  single  chip  implemen¬ 
tation  of  the  radar  emitter  feature  data.  What  perhaps  is  not  obvious  is  that  as  the 
number  of  classes  increase,  so  does  the  total  error.  As  the  number  of  training  epochs 
increase,  the  total  error  decreases.  The  ETANN  test  results  worsen  as  the  number 
of  classes  increase  for  this  particular  problem.  This  trend  suggests  the  ETANN  does 
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Table  11.  Training  and  Testing  Results  III. 
75  Training  Vectors  Per  Class  (2250  Total) 
and  25  Test  Vectors  Per  Class  (750  Total) 


Iterations 

(xlOOO) 

Classes 

Network  Size 
( input-hidden-output ) 

Total 

Error 

Neural  Graphics 
Simulator 
(Train/Test) 

ETANN 
(Test  Only) 

mSSM 

MM 

16x16x30 

48 

82%/81% 

76% 

HI 

■■ 

16x20x30 

44 

86%/85% 

81% 

200 

30 

16x25x30 

42 

77% 

200 

30 

16x30x30 

41 

91%/91% 

72% 

200 

30 

16x35x30 

41 

92%/92% 

83% 

not  have  enough  bit  resolution  to  separate  all  the  classes  with  the  simulator  trained 
weights.  The  classification  results,  as  with  all  results  in  this  chapter,  are  an  average 
of  three  measurements. 

Perhaps  the  problem  cannot  be  solved  to  user  satisfaction  using  a  single  chip 
with  weight  resolution  of  7-8  bits  without  resorting  to  chip-in-loop  (CIL)  training. 
The  primary  reason  for  having  characterized  the  ETANN  sigmoidal  transfer  function 
in  the  first  place  was  to  limit  the  amount  of  CIL  training  required,  because  CIL 
training  decreases  the  useful  life  of  the  ETANN  (3:19-8).  Due  to  limited  time,  a 
CIL  program  was  not  written,  but  it  is  recommended  for  future  research.  With 
considerable  effort,  it  is  possible  to  code  a  C  routine  to  perform  CIL  training.  After 
10  to  20  epochs  of  CIL  training,  classification  accuracy  should  be  improved.  CIL 
compensates  for  chip  variance  and  loss  of  resolution  when  going  from  simulator  to 
chip. 


4-3.2  Multi-Chip  Testing  Using  the  EMB  The  ETANN  Multi-chip  Board 
(EMB)  is  a  platform  that  allows  a  designer  to  implement  up  to  eight  chips  in  var¬ 
ious  configurations  for  neural  computing.  The  EMB  was  used  in  this  research  to 
implement  the  primary  thesis  problem  solved  in  simulation  (5).  The  best  training 
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Table  12.  TVaining  and  Testing  Results  IV. 

75  TVaining  Vectors/Class  (2250  Total) 
and  25  Test  Vectors  Per  Class  (750  Total) 


Number  of 
Epochs 

Total 

Error 

Network 

Size 

Simulator 

Training 

Simulator 

Test 

ETANN 

Test 

44 

25 

16x35x30 

95% 

93% 

75% 

89 

24 

16x35x30 

96% 

94% 

83% 

133 

24 

16x35x30 

96% 

95% 

81% 

177 

24 

16x35x30 

96% 

95% 

82% 

results  achieved  in  (5)  came  from  using  a  hierarchical  system  of  classifying  30  radar 
emitters.  The  hierarchical  approach  which  gave  the  best  test  results  in  simulation 
is  implemented  in  ETANN  hardware  using  three  devices  on  the  EMB  platform  for 
performance  evaluation  and  simulator  comparison.  First,  a  brief  description  of  the 
hierarchical  approach  is  presented,  and  then  the  hardware  results  are  shown. 

4.3.2.I  Hierarchical  Approach  As  described  in  (5),  the  hierarchical  ap¬ 
proach  divides  a  large-class  network  into  smaller  networks  with  one  or  more  group¬ 
ing  networks.  For  example,  if  one  were  to  divide  a  30-class  problem  into  two  sub¬ 
networks,  a  total  of  three  networks  would  be  needed  to  perform  the  classification 
task.  Suppose  the  networks  for  the  hierarchy  used  in  research  are  listed  as  follows: 

1.  A  2-class  network  for  separating  the  30  classes  into  two  groups — used  as  a 
switch  to  control  the  other  two  networks. 

2.  A  12-class  network  (bottom  12)  to  classify  group  one. 

3.  An  18-class  network  (top  18)  to  classify  group  two. 

This  hierarchy  is  but  only  one  example  of  the  infinite  number  of  possible  designs. 
It  just  so  happens  that  this  design  is  the  network  hierarchy  used  in  this  research. 
Figure  11  shows  a  block  diagram  for  the  hierarchical  approach  used  in  this  research. 
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Network  two  acts  as  a  switch  on  the  output  that  selects  one  of  two  classes;  one  class 
contains  all  the  bottom  12  classes,  and  the  other  contains  all  the  upper  18  classes. 

4. 3. 2.2  Hierarchical  Configuration  Results  Until  now  most  of  the  previ¬ 
ous  experiments  were  performed  on  a  single  chip,  discounting  chipnto-chip  differences. 
In  the  following  experimental  results,  chip-to-chip  variances  were  investigated  for  a 
multi-chip  implementation.  Six  chips  were  used  in  the  experiments.  On-chip  and 
chip-to-chip  variance  test  results  are  in  Tables  13  and  14. 

Table  13  shows  the  on-chip  variance  test  results  for  six  chips  in  three  different 
networks.  The  networks  listed  in  this  table  are  one  example  of  a  hierarchical  network 
used  to  solve  the  radar  emitter  problem.  To  compute  a  standard  deviation,  three 
ETANN  tests  were  performed  on  every  chip,  and  the  result  was  averaged.  Variation 
on  a  single  chip  is  practically  negligible. 

On  the  other  hand,  chip-to-chip  variation  as  shown  in  Table  14  is  more  signifi¬ 
cant,  averaging  about  ±3  percent  for  all  the  networks  shown.  This  finding  reinforces 
the  need  for  chip-in-loop  training,  which  was  not  performed  in  this  research  because 
of  limited  time.  The  data  used  to  train  the  hierarchical  networks  were  partitioned 
as  follows:  1)  Network  One:  2250  training  vectors  (18/12  split  into  two  classes), 
2)  Network  Two:  900  training  vectors  (bottom  12  classes)  ,  and  3)  Network  Three: 
1350  training  vectors  (top  18  classes).  The  number  of  training  iterations  for  each 
network  was  as  follows:  1)  Network  One:  600k,  2)  Network  Two:  500k,  and  3) 
Network  Three:  400k. 

Table  15  shows  the  final  hierarchical  implementation  results  for  three  different 
network  sizes.  The  hierarchical  probability  was  computed  using  the  actual  results 
froir  'r  chip  tests.  The  simulator  tests  for  the  hierarchical  network  are  the  same  as 
those  listed  in  Table  14. 

Without  chip-in-loop  training,  it  appears  that  the  best  an  ETANN  device  can 
do  with  this  particular  data  (30-classes)  is  about  87  percent  classification  rate.  In- 
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Figure  11.  Block  Diagram  for  Hierarchical  System 
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Table  13.  On-Chip  Variance  Test  Results 


Network 

Simulator 

Total 

Size 

Train/Test 

Error 

16x20x2 

99%/96% 

23 

>1 

11 

11 

11 

11 

11 

11 

W 

11 

11 

91 

11 

11 

16x20x12 

97%/96% 

18 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

16x20x18 

99%/98% 

19 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

11 

Table  14.  Chip-to-Chip  Variance  Test  Results 


Network 

Size 


Netl  16x20x2 
Net2  16x20x12 
Nets  16x20x18 


Netl  16x25x2 
Net2  16x25x12 
Net3  16x25x18 


Netl  16x30x2 
Net2  16x30x12 
Nets  16x30x18 


Simulator  Test  Avg.  of  Combined 
Test  6  Chips  Deviation 


±1.3% 

±3.6% 

±4.0% 
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creasing  the  number  of  hidden  nodes  did  not  improve  performance  for  the  hierarchi¬ 
cal  networks  as  it  did  in  single  chip  implementations  (assuming  the  same  number  of 
training  iterations),  but  it  did  decrease  the  amount  of  deviation  from  the  average 
result.  More  training  might  cause  a  network  with  more  hidden  neurons  to  improve 
classification  accuracy,  or  it  might  cause  the  network  to  generalize  poorly.  But  from 
observation  of  the  weight  file,  the  majority  of  the  weights  were  near  their  maximum 
values  that  they  could  take  on  (an  indication  that  the  network  was  near  “brain 
dead”). 

The  term  “brain  dead”  is  used  to  describe  a  network  that  will  not  train  any 
longer.  If  the  network  is  near  some  acceptable  range  of  classification,  then  chip>-in- 
loop  (on-chip)  training  should  be  all  that  is  needed  to  train  the  network  to  some 
acceptable  error.  If  the  network  is  “brain  dead”  and  the  network  is  still  performing 
unacceptably,  then  try  restarting  training  with  a  new  set  of  random  weights. 

The  drawbacks  to  using  a  hierarchical  approach  are  added  complexity,  addi¬ 
tional  number  of  devices,  and  increased  processing  time  (about  3/isec  for  this  hier¬ 
archy).  Furthermore,  there  are  no  guarantees  of  better  performance  over  a  single 
chip  implementation.  Even  though  in  this  training  example  the  on-chip  test  results 
were  better  for  the  hierarchical  networks  than  for  the  single  chip  networks,  the  sin¬ 
gle  chip  networks  performed  better  in  simulation  (compare  simulation  results  in  the 
tables).  Because  of  resolution  loss  when  going  from  simulator  to  chip  with  higher- 
order  networks,  on-chip  tests  will  be  poorer  for  single  chip  implementations  than  for 
hierarchical  (multi-chip)  implementations  of  the  same  network  problem. 

4. 3. 2.3  Research  Sub-Objective  Findings  The  primary  objective  of  this 
research  was  to  implement  an  artificial  neural  network  in  the  ETANN  hardware  to 
classify  radar  emitter  data.  The  sub-objectives  in  this  research  include  the  following: 

1.  Minimize  the  processing  time  and  maximize  the  classification  accuracy. 

2.  Develop  a  technique  to  load  training  weights  from  Neural  Graphics  to  iNNTS. 
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Table  15.  Hierarchical  Implementation  Results 


Hierarchical 

Network  Size  Architecture 

Hierarchical 

Probability 

Combined 

Deviation 

16x20x2  /  16x20x12  /  16x20x18 

87% 

±9.5% 

16x25x2  /  16x25x12  /  16x25x18 

84% 

±7.5% 

16x30x2  /  16x30x12  /  16x30x18 

86% 

±8.9% 

3.  Estimate  processing  time  to  make  a  classification  in  hardware. 

4.  Compare  resolution  differences  between  the  hardware  device  and  simulator 

5.  Calculate  an  implementation  loss  factor. 

To  minimize  processing  time  yet  maximize  classification  accuracy  requires  a 
designer  to  consider  trade-offs.  The  best  on-chip  classification  accuracy  came  from 
using  a  hierarchical  implementation;  however,  the  fastest  network  was  a  single  chip 
implementation  using  a  two  layer  (inputs  x  hidden  x  outputs)  network. 

A  method  of  transferring  the  Neural  Graphics  trained  weights  to  iNNTS  was 
accomplished  with  success.  Appendix  H  contains  the  C  tools  necessary  to  accomplish 
this  objective. 

Processing  time  is  dependent  on  network  architecture.  This  means  as  more 
layers  are  embedded  in  the  network,  the  longer  the  time  it  will  take  to  process  the 
feature  information  and  produce  an  output.  Each  layer  of  a  network  takes  3  /isec  to 
process  input  data  and  produce  an  output.  For  the  single  chip  implementation  to 
the  thesis  problem  (assuming  two  layers),  the  processing  time  takes  6  fisec.  For  the 
hierarchical  network  in  this  research,  however,  the  processing  time  takes  a  minimum 
of  9  fjsec — 6  fisec  for  three  independent  networks  (on  separate  chips)  to  process  the 
feature  data  simultaneously  and  3  /xsec  to  select  the  winning  output. 

According  to  (20),  chip  weight  resolution  is  about  7.5  bits  (about  three  decimal 
places),  and  device  processing  resolution  is  about  6.5  bits  (between  two  and  three 
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decimal  places).  The  Neural  Graphics  simulator  uses  more  than  32  bits  of  resolu¬ 
tion  in  processing  numerical  data  (six  decimal  places).  The  trained  weights  from 
Neural  Graphics  were  rounded  off  at  eight  bits  resolution  (three  decimal  places). 
For  the  data  used  in  this  research,  the  ETANN  chip  accuracy  began  dropping  off 
with  eight-class  networks  and  was  severely  degraded  with  26-class  networks  (in  early 
experiments).  After  using  improved  training  methods  (in  section  4.3),  accuracy 
degradation  was  reduced  to  more  acceptable  levels  for  up  to  thirty-class  networks — 
fair  results  considering  that  no  chip-in-loop  training  was  used  to  compensate  for 
chip-to-chip  variance  or  loss  of  resolution. 

The  loss  of  resolution  when  going  from  simulator  to  chip  is  network  depen¬ 
dent,  but  was  more  pronounced  in  single  chip  implementations  than  in  hierarchical 
networks  for  the  same  network  problem.  Hierarchical  networks  lower  the  loss  of  res¬ 
olution  by  breaking  large-class  problems  into  sub-networks  with  fewer  classes.  The 
fewer  the  number  of  classes,  the  less  the  loss  of  resolution  when  going  from  simulator 
to  chip.  The  reason  is  because  of  network  complexity.  It  takes  less  weight  resolution 
to  separate  fewer  classes,  but  as  the  amount  of  training  data  increases,  so  does  the 
required  weight  resolution. 

The  implementation  loss  computed  for  this  research  assumed  no  chip-in-loop 
(on-chip)  training.  For  a  single  chip  network  implementation  (16  x  35  x  30),  the 
average  (of  four  networks)  implementation  loss  per  chip  was  14  percent.  For  the 
hierarchical  network,  the  average  (of  nine  networks)  implementation  loss  per  chip  was 
3  percent,  but  the  hierarchy  consists  of  three  chips — the  total  average  implementation 
loss  equaled  9  percent.  Again,  implementation  loss  is  a  factor  of  network  size,  data 
complexity,  amount  of  data,  and  chip  variance.  Loss  factors  will  be  different  for 
other  network  implementations. 
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4.3.3  Feature  Saliency  The  Neural  Graphics  program  provides  a  measure  of 
the  importance  of  a  particular  feature  in  the  network  classification  process,  and  it  is 
called  saliency  (5:29). 

Cameron  used  five  data  sets  in  25  runs  to  determine  the  saliency  of  each  feature. 
He  then  calculated  average  and  final  ranking  of  each  feature  over  all  the  runs — see 
Table  16. 

•  Classes:  30 

•  Training  vectors:  50  per  class 

•  Test  vectors:  25  per  class 

•  Hidden  nodes:  20 

•  Iterations:  variable;  10,000  to  30,000 

•  Special  note:  Neural  Graphics  toggled  Saliency  On 

The  most  important  feature  for  the  classification  process  was  feature  16;  how¬ 
ever,  there  was  not  a  significant  drop  in  network  accuracy  when  feature  16  was 
omitted.  This  result  could  be  due  to  the  redundancy  of  information  contained  in 
the  other  15  features.  The  other  features  were  less  important;  there  were  no  large 
discontinuities  in  the  ranking  (5:30). 

4.4  Conclusion 

This  chapter  presented  findings  on  ETANN  sigmoidal  transfer  function  tests 
and  on  single  chip  and  multi-chip  experiments.  Test  results  showed  that  the  ETANN’s 
sigmoidal  transfer  function  can  be  accurately  characterized  through  simple  experi¬ 
ments. 

The  thesis  problem  and  sub-objectives  were  met  with  little  difficulty  once  the 
Neural  Graphics  simulator  was  successfully  modified  to  emulate  the  ETANN  device 


55 


Table  16.  Feature  Saliency  Ranking  (5:30) 


Feature 

Avg.  Rank 

Ranking 

4.4 

2 

11.2 

12 

12.6 

13 

10.5 

11 

5 

8.3 

9 

6 

7.1 

6 

7.5 

8 

5.6 

5.0 

10 

7.2 

11 

12.8 

12 

14.0 

13 

14.2 

14 

10.1 

15 

4.5 

16 

1.0 

1 

and  a  method  of  porting  the  weights  out  of  Neural  Graphics  into  iNNTS  was  devel¬ 
oped.  The  maximum  on-chip  classification  accuracy  for  a  single  chip  implementation 
of  the  30-class  problem  without  chip-in-loop  training  was  83  percent,  using  35  hidden 
neurons.  Again  without  chip-in-loop  training,  the  maximum  on-chip  classification 
accuracy  for  a  hierarchical  configuration  with  the  30-class  problem  was  87  percent. 
The  total  average  implementation  loss  was  found  to  be  9  percent  for  the  hierarchical 
network. 

FVom  a  feature  saliency  experiment,  the  most  important  feature  found  in  the 
classification  process  was  feature  16,  and  the  least  important  was  feature  13.  For 
security  reasons,  the  features  of  the  data  are  only  known  as  radar  emitter  character¬ 
istics. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


5.1  Conclusions 

Results  of  this  reseaxch  are  ground-breaking  in  several  categories.  First  of  all, 
no  radar  emitter  classification  problem  has  been  implemented  in  ETANN  hardware 
until  now.  Also,  there  have  been  no  reports  of  using  customized  simulators  on  work¬ 
stations  to  train  networks  for  the  ETANN  chip.  Furthermore,  there  have  been  no 
published  reports  of  the  ETANN  chip’s  sigmoidal  transfer  function  being  character¬ 
ized  for  non-binary  operation.  Below  is  a  summary  of  the  significant  contributions 
in  this  research: 

1.  Characterized  the  ETANN  sigmoidal  transfer  function.  The  ETANN  sigmoidal 
transfer  function  was  not  previously  characterized;  now  it  has  been  accurately 
characterized  and  tested  in  a  custom  simulator.  Neural  Graphics. 

2.  Customized  the  Neural  Graphics  simulator  for  ETANN  emulation.  Neural 
Graphics  was  developed  primarily  for  emulating  the  Cybenko  type  of  network 
using  1/(1  -f  e"®)  as  a  sigmoid  function.  Now,  Neural  Graphics  can  emulate 
the  ETANN  chip  using  a  modified  tanh  function,  which  was  derived  during 
experiments  on  the  ETANN  chip. 

3.  Developed  interface  software  with  iNNTS.  At  the  beginning  of  this  research 
there  was  no  software  to  interface  with  iNNTS  for  the  purpose  of  writing  weight 
values,  which  were  trained  on  a  custom  simulator,  to  the  ETANN  chip.  In 
addition,  there  was  no  software  for  testing  the  on-chip  classification  accuracy. 
Four  C  programs  were  written  to  interface  with  iNNTS — two  programs  each  for 
the  single  chip  and  the  ETANN  Multi-chip  Board  (EMB)  modes  of  operation. 

4.  Implemented  thesis  problem  in  single  chip  and  multi-chip  configurations.  Until 
now  the  30-class  radar  emitter  identification  problem  had  only  been  solved  in 
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simulation  using  a  Cybenko  type  sigmoidal  transfer  function  on  Neural  Graph¬ 
ics  (5). 

5.  Defined  feature  saliency  for  thesis  problem.  Feature  saliency  was  defined  pre¬ 
viously  in  (5)  and  was  accomplished  a  second  time  during  this  thesis  research. 
The  results  are  the  same. 

For  the  purpose  of  emulating  an  ETANN  device  in  a  custom  simulator,  a 
sigmoidal  transfer  function  was  derived,  tested,  and  implemented  as  a  model  for 
yoAiN  ~  3.3V,  64-input  operation.  The  result  is  the  following  equation: 


1  83 

JVeuron  Outtmt  =  ^  ^  '  -9“  (9) 

Neural  Graphics,  am  AFIT  neurad  network  simulator  running  on  a  Silicon 
Graphics  (model  4D)  workstation,  was  customized  for  use  with  the  ETANN  chip. 
Customizing  Neural  Graphics  required  changing  the  coded  sigmoidal  transfer  func¬ 
tion,  coding  a  limitation  routine  on  weight  values  to  keep  weight  values  below  the 
ETANN’s  maximum  value,  adding  more  bias  weight  values  for  chip  emulation,  aind 
modifying  Neural  Graphics’  “makeinput.c”  file  to  optimize  desired  and  non-desired 
training  values  in  the  tradning  (i.e.,  in  general  a  simulator  coded  with  the  tanh  trans¬ 
fer  function  should  train  to  -1  for  non-desired  class  and  +1  for  desired  class). 

In  addition  to  modifying  the  sigmoidal  transfer  function,  there  were  several 
other  ETANN  emulation  problems  to  overcome  in  Neural  Graphics.  For  instance,  the 
weight  values  in  Neural  Graphics  grew  steadily  as  the  number  of  training  iterations 
increased,  and  thus  they  exceeded  the  ETANN  chip’s  dynamic  range  in  the  course  of 
training.  Moreover,  the  unmodified  version  of  Neural  Graphics  trains  a  network  using 
a  single  bias  weight  value  per  neuron,  whereas  the  ETANN  chip  may  use  up  to  seven 
bias  weight  values  per  neuron.  The  additional  neurons  give  added  dynamic  range 
to  the  weight  dimension  space,  which  makes  simulator  and  chip  training  easier.  In 
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another  problem  the  Neural  Graphics  weight  values  were  not  written  in  a  compatible 
format  for  programming  in  iNNTS.  The  solution  to  this  problem  required  developing 
software  to  interface  the  Neural  Graphics’  weight  file  with  iNNTS  and  writing  the 
weight  values  to  a  chip,  both  in  single-chip  and  ETANN  multi-chip  board  (EMB) 
modes  of  operation.  Furthermore,  a  classification  program  was  written  to  classify 
test  vectors  on  the  ETANN  chip.  All  of  these  problems  were  solved  during  the  first 
phase  of  research  (covered  in  Chapter  4). 

In  the  second  phase  of  research,  emphasis  was  on  solving  the  radar  emitter 
identification  problem  and  the  sub-objectives.  The  maximum  on-chip  classification 
accuracy  for  a  single  chip  implementation  without  chip-in-loop  training  was  83  per¬ 
cent,  using  35  hidden  neurons  (16  x  35  x  30).  Again  without  chip-in-loop  training, 
the  maximum  on-chip  classification  accuracy  for  a  hierarchical  configuration  with 
the  30-class  problem  was  87  percent,  using  20  hidden  neurons  (chip  one  [16  x  20  x 
2],  chip  two  [16  X  20  X  12],  and  chip  three  [16  x  20  x  ICj). 

Fast  operation  (speed),  in  general,  is  a  major  consideration  for  emitter  classi¬ 
fication  in  hardware.  The  highest  on-chip  classification  accuracy  from  these  experi¬ 
ments  came  from  using  a  hierarchical  implementation;  however,  the  fastest  network 
came  from  using  a  single  chip  implementation  of  two  layers  (input  x  hidden  x 
output).  The  speed  versus  classification  accuracy  trade-off  is  a  common  factor  in 
simulation  as  well  as  in  hardware  implementation. 

A  method  of  transferring  Neural  Graphics’  trmned  weights  to  iNNTS  was  ac¬ 
complished  by  formatting  the  weight  values  in  an  ASCII  file  and  then  running  an 
interface  program  with  iNNTS  to  write  the  weights  to  a  chip.  Developing  this  method 
was  essential  to  implementing  a  neural  network  in  hardware  using  the  customized 
Neural  Graphics  simulator. 

Implementation  loss  factors  are  network  and  data  dependent.  For  the  hierar¬ 
chical  network  investigated  in  this  research,  the  total  average  implementation  loss 
(from  three  experiments)  was  found  to  be  9  percent.  In  other  words,  one  can  ex- 
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pect  to  lose  classification  accuracy  by  approximately  9  percent  when  implementing 
a  device  with  the  Neural  Graphics  simulator  trained  weights,  without  chip-in-loop 
(on-chip)  training,  for  this  particular  data. 

The  last  experiment  was  to  determine  feature  saliency.  Chapter  4  shows  the 
saliency  results  with  the  features  in  rank  order.  The  most  important  feature  for  the 
classification  process  was  feature  16,  and  the  least  important  was  feature  13.  For 
security  reasons,  the  features  of  the  data  are  only  known  as  radar  emitter  character¬ 
istics. 

5.2  Recommendations 

Topics  in  this  thesis  could  be  explored  further.  The  single  most  important  topic 
to  continue  research  is  on  development  of  a  Chip-In-Loop  (CIL)  training  program 
using  the  iNNTS  software  interface.  With  a  CIL  capability,  one  can  finish  training 
the  simulator-trained  chips  (as  done  in  this  research)  to  an  acceptable  error  rate. 

Another  topic  to  continue  research  on  is  the  use  of  different  criteria  for  the 
partitioning  of  the  classes  for  hierarchical  systems.  Perhaps  one  could  yield  further 
increases  in  classification  accuracy.  If  both  of  these  recommendations  are  followed, 
increases  in  data  and  class  size  can  be  investigated  and  the  results  compared  to  those 
in  this  thesis. 

5.3  Summary 

This  research  has  demonstrated  that  artificial  neural  networks  can  be  used  in 
the  design  of  radar  emitter  identification  systems  and  be  implemented  in  hardware 
with  acceptable  accuracy  (considering  no  chip-in-loop  training)  for  30  classes.  It 
has  also  been  demonstrated  that  the  hierarchical  approach  to  the  30-class  problem 
results  in  a  greater  on-chip  accuracy  than  that  of  a  single  chip  implementation.  The 
maximum  on-chip  classification  accuracy  for  a  single  chip  implementation  was  83 
percent.  For  a  multi-chip  implementation  using  a  hierarchical  approach,  the  maxi- 


mum  on-chip  classification  accuracy  was  87  percent.  The  drawbacks  to  a  hierarchical 
approach,  however,  are  added  complexity,  added  number  of  devices,  and  added  pro¬ 
cessing  time  (about  3/zsec  for  this  hierarchy).  For  the  hierarchical  network,  the 
total  average  implementation  loss  was  found  to  be  9  percent  in  this  problem.  It 
is  recommended  that  a  chip-in-loop  training  interface  with  iNNTS  be  developed  in 
future  research  efforts.  And  finally,  investigation  into  the  use  of  different  criteria 
for  partitioning  the  classes  for  hierarchical  systems  could  yield  further  increases  in 
classification  accuracy.  Thus,  following  these  recommendations  would  allow  further 
research  efforts  in  ETANN  hardware  implementation  with  larger  class  problems  than 
those  studied  here. 
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Appendix  A.  Partial  Derivative  of  ETANN  Sigmoidal  Transfer 

Function 


This  appendix  presents  a  derivation  for  the  partial  derivative  of  the  general 
type  of  sigmoid  function  used  in  the  ETANN  chip.  The  vanilla  back  propagation 
algorithm  as  described  by  Rogers  and  others  (28)  uses  a  unipolar  sigmoid  function, 
sometimes  referred  to  as  a  squashing  or  logistic  function.  Below  is  a  derivation  of 
the  partial  derivative  for  the  general  sigmoid  function  used  in  the  ETANN  device, 
sometimes  referred  to  as  a  bipolar  or  symmetric  sigmoid. 
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For  the  ETANN  sigmoidal  transfer  function  defined  in  Chapter  4,  Ekjuation  5, 
it  can  be  easily  shown  that  the  derivative  results  in  the  following  equation: 

df  (1.74a)  _  +  -  .94} 

da  da 

=  .95[.94  +  /(1.74o)][.89  -  /(1.74q)] 


Recall  that  for  backward  error  propagation  the  weights  are  updated  as  follows; 
wj  =  -  TfSjXi  +  a(w^j  -  w-j) 

where  to.j  is  the  weight  from  node  i  to  node  j  in  the  next  layer,  x,-  is  the  output  of 
node  i,  and  Sj  is  the  error  associated  with  node  j.  The  t)  and  q  represent  learning 
rates.  The  new  weight  is  represented  by  the  current  weight  by  and  the  old 
weight  by  w^.  Thresholds  are  adapted  similarly  where  x,  is  replaced  by  +1  if  the 
threshold  is  ^ded  to  the  weighted  sum  and  —1  if  it  is  subtracted. 

The  Sj  for  the  ETANN  equations  are  defined  in  this  research  as  follows; 

X  _  /  •95[.94  4-  2/j][.89  —  —  j/y)  for  output  node  j 

^  ~  \  .95[.94  +  xy][.89  -  xy]  SkWjk  for  hidden  node  j 

where  xy  and  yy  =  /(1.74q)  in  the  hidden  and  output  nodes  respectively;  dy  is  the 
desired  output  for  output  node  j,  and  yy  is  the  actual  output.  For  the  hidden  nodes 
the  Sk  are  the  errors  for  the  layers  above  (28;55). 
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Appendix  B.  How  to  Make  an  iBrainmaker  Weight  File 


B.l  Weight  File  Format 

The  file  format  for  the  weights  file  is  best  shown  through  an  example.  Figure 
12  shows  an  example  of  a  network  with  weighted  interconnections  between  input 
and  hidden  neurons  (nodes)  and  hidden  and  output  neurons  (nodes).  This  example 
illustrates  only  one  hidden  layer;  there  may  be  any  number  of  hidden  layers,  but 
it  has  been  shown  that  one  hidden  layer  suffices  (with  enough  hidden  nodes)  to 
characterize  any  function  (28:52).  The  number  of  hidden  nodes,  however,  is  problem 
dependent. 


The  format  for  writing  the  weights  to  a  file  is  shown  below.  Note  that  w_ll 
represents  the  weight  from  input  node  number  one  to  hidden  node  number  one.  The 
weights  are  written  from  layer  one  (input  to  hidden  layer)  to  the  output  layer.  The 
file  looks  something  like  the  following: 
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weights 

v.ll  w_21  bias  (weights  to  hidden  neuron  #1) 
w_12  w_22  bias  (weights  to  hidden  neuron  #2) 


w_14  w_24 
(empty  line 
w.ll  W.21 
w_12  w_22 


bias  (weights  to  hidden  neuron  #4) 
between  layers) 

w_31  w_41  bias  (weights  to  output  neuron  #1) 
w_32  w_42  bias  (weights  to  output  neuron  #2) 


Weights  take  on  a  range  of  -2.5  to  2.5  for  ETANN  configuration.  Input  values 
are  limited  in  the  range  of  -1.0  to  -1-1.0.  The  number  of  bias  weights  per  neuron  is 
user  defined  (usually  one  to  seven  bias  weights  per  neuron  are  used  on  the  ETANN 
chip). 


B.2  Writing  Weights  in  Neural  Graphics 

Neural  graphics  can  be  modified  to  write  your  weights  into  a  file  of  any  desired 
format  by  simply  modifying  the  ’’saver.c”  and  ’’general.c”  files.  An  example  of  mod¬ 
ifying  ”  saver .c”  and  ’’generzd.c”  is  shown  below.  These  files  were  modified  to  write 
the  weights  into  a  file  named  ’’weights.calvin”  in  the  same  format  as  iBrainMaJcer. 


*  saver.c  modification 


* ’ll  it  4c  4c  *))(*#*****  itc  )|I  **  / 

/*  section  added  to  saver.c  file  ♦/ 
write_calvin_weights (filename) 
char  filenameC]; 


FILE  ♦fpx; 
int  ij.k; 

fpx  =  fopen(filename,"w'') ; 
if  (fpx  ==  NULD-C 

printf ("Couldn’t  open  weight  file  "); 
ggexitO; 


} 
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fprintfCfpx, "weights  \n  “); 
loopi  (parameters. layers ){ 
int  sizex; 

loopj  (net->layerl [i] “>size_output) { 

loopk(net->layerl [i] ->size_input) 
fprintf(fpx,“5lf  ",net->layerl[i3->weights[j]  [k]); 
fprintf  (fpx, "  7.f  “,net->layerl  [i]->theta[j] ) ; 

fline; 

} 

fline; 

} 

fclose(fpx) ; 

} 

/♦♦♦♦♦♦♦♦♦****************,„*^^^^^****^*^^^^^^^^^^^^^^^ 

*  general. c  modification 

for(avg=l;avg  <=avgs  ;avg++)-C 

check_parameter_line(argc,argv) ; 

DO.NETWORKO ; 
hold_one.out() ; 
f ile.saliency(O) ; 
do.avgO ; 

/♦  following  line  added  to  code  in  general. c  file  */ 
write.calvin.weights ("weights . calvin") ; 
write.weightsC'weights.temp") ; 

#ifdef  TERMINAL 
endwinO ; 

#endif 
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Appendix  C.  How  to  Design  a  Network  for  the  ETANN  Chip 


C.l  Overview  of  Design  Process 

Design  of  an  ANN  is  basically  a  seven-step  process  of  working  through  a  log¬ 
ical  description  of  the  problem  to  be  solved.  First  of  all,  decide  whether  the  ANN 
is  to  predict,  generalize,  or  recognize  (34).  Some  examples  include  stock  market 
average  prediction,  data  generalization  for  loan  risk  analysis,  and  automatic  target 
recognition  systems  using  multi-sensor  input  to  recognize  tanks,  trucks,  and  jeeps. 

Next,  decide  what  information  will  make  good  features  for  predictions,  gener¬ 
alizations,  or  recognitions.  Predictions  are  based  on  past  information  of  pertinent 
data.  For  example,  in  predicting  stock  averages,  information  to  be  used  for  predic¬ 
tion  might  include  previous  stock  prices,  market  interest  rates,  state  of  the  economy, 
and  world  market  averages,  to  name  a  few.  Good  features  make  good  classifiers 
(predictors).  The  selection  of  information  to  use  as  features  is  more  an  art  than  a 
science.  Follow  your  logical  intuition  and  experiment  with  various  types  of  data. 

Following  this  last  step,  you  will  need  to  get  lots  of  data.  Generally,  the  more 
data,  the  better  the  predictor,  generalizer,  or  recognizer.  One  rule  states  that  using 
a  minimum  of  three  times  the  number  of  input  features  will  give  a  rough  estimate 
of  how  many  training  vectors  are  needed  per  class  (Foley’s  rule)  (28:60).  If  the  total 
number  of  training  vectors  is  less  than  twice  the  number  of  features,  the  training 
data  error  rate  will  be  near  zero  regardl^  of  the  distribution,  which  is  a  result  of 
Cover’s  theorem  on  capacity  (28:60). 

When  building  a  feed-forward  network,  there  are  two  things  needed  to  make 
the  network.  One  needs  samples  of  input  data  and  to  know  the  desired  results. 
The  next  step  is  to  build  the  network.  Two  things  are  needed  to  make  a  network:  a 
network  definition  and  a  collection  of  data.  A  network  definition  consists  of  assigning 
a  number  of  input  nodes  to  a  number  of  hidden  nodes  to  a  number  of  output  nodes, 
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see  Figure  13.  The  number  of  inputs  in  Figure  13  is  two,  the  number  of  hidden 
neuron  nodes  is  two,  and  the  number  of  outputs  is  one.  The  assignment  of  input 
nodes  and  output  nodes  is  problem  dependent.  The  assignment  of  hidden  nodes 
is  problem  and  data  dependent.  If  the  problem  is  linearly  separable,  then  hidden 
nodes  are  not  necessary,  unless  there  is  an  enormous  number  of  training  facts.  In 
that  case  add  a  hidden  layer  to  improve  network  performance.  Assign  the  number 
of  hidden  nodes  to  correspond  to  the  network  complexity  (linearly  separable  versus 
non-linearly  separable)  by  trial  and  error  results.  The  goal  is  to  train  effectively  with 
as  few  hidden  nodes  as  possible,  so  as  to  minimize  the  number  of  computations  in 
the  network.  Additional  layers  of  hidden  nodes  may  become  necessary  if  the  network 
is  quite  large  and  has  a  lot  of  training  facts. 

After  building  the  network  comes  training  the  network.  Code  your  favorite 
feed-forward  training  algorithm  or  use  off-the-shelf  programs  like  iBrainMaker  (3)  or 
iDynaMind  (23),  which  both  use  the  popular  back  propagation  training  algorithm. 

Training  a  network  is  also  more  an  art  than  a  science.  Adjusting  learning 
parameters  such  as  momentum,  learning  rate,  and  tolerance  all  become  factors  in 
how  fast  a  network  will  converge  (within  some  acceptable  error  range). 

Testing  the  network  follows  training  the  network.  Always  save  a  percentage 
of  the  data  for  testing.  Generally,  10-20  percent  of  the  total  number  of  vectors  is 
sufficient  to  test  the  network.  It  is  important  to  show  the  network  data  that  it  has 
not  seen  before,  because  only  then  will  one  know  if  the  network  is  a  good  predictor, 
generalizer,  or  recognizer.  Finally,  good  results  will  depend  on  the  network  definition 
and  comparisons  with  other  classifying  techniques. 

The  last  step  in  this  process  is  running  the  network,  presenting  it  with  new 
input  data,  and  gathering  results.  The  firet  two  steps  above  are  the  most  difficult 
steps  in  the  design  process.  As  the  saying  goes,  ”  A  problem  well-defined  is  a  problem 
half  solved.” 
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C.2  Summary 

•  Step  1.  Define  the  problem.  Decide  whether  you  want  the  ANN  to  predict, 
generalize,  or  recognize. 

•  Step  2.  Represent  the  Information.  Decide  what  information  will  make  good 
features  for  predictions,  generalizations,  or  recognitions. 

•  Step  3.  Get  lots  of  data. 

•  Step  4.  Define  the  network.  Define  the  number  of  inputs,  hidden  nodes, 
outputs,  and  training  data. 

•  Step  5.  Train  the  network. 

•  Step  6.  Test  the  network. 

•  Step  7.  Run  the  Network. 
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Appendix  D.  How  to  Convert  GTRI  Data  to  Simulator  Formats  and 

Shuffle 


D.  1  Introduction 

What  follows  below  is  the  C  code  used  in  this  research  for  data  conversion 
and  vector  shuffling.  Data  conversion  from  GTRI  format  to  iBrainMaker  and  Neural 
Graphics  formats  was  necessary  for  thesis  research.  The  shuffling  programs  are  used 
to  randomize  the  data  vector  order  before  training  and  testing  a  neural  network. 

D.2  How  to  Convert  GTRI  Data  to  iBrainMaker  Format 

Purpose:  This  program  reads  a  GTRI  data  file  and  creates 
an  iBrainMaker  fact  file  for  processing  in  iBrainMaker. 

This  particular  program  is  limited  to  three  classes,  but  it 
is  easy  to  extend  the  ideas  here  for  a  file  of  several 
classes. 

Date:  5  June  1992 
Author:  Capt  James  Calvin 


#include<stdio . h> 


/*==*============== 

Begin  Main 


mainO 

FILE  *in_ptr,  *out_ptr; 
int  X,  j,  dimension; 
int  vector [20] ; 

int  class,  junkl,  junk2,  junkS,  junk4; 

int  no.of .vectors ; 

int  temp.l; 

dimension  =  17; 

no.of .vectors  =  5795; 


71 


/♦*===»============= 

Open  Files 


in.ptr  =  fopenC'allSO.d", "r") ; 
out.ptr  =  fopen("3-class_bm.dat'',"w‘') ; 

f printf (out. ptr, "Line#  fl  f2  f3  f4  f4  f6  f7  f8  f9 
flO  fll  fl2  fl?  fl4  fl5  fl6  Cl  C2  C3  \n"); 

fscanf  (in_ptr,"*/,d  */,d  */,d  */,d  \n",tjunkl,tjunk2,tjunk3,tjunk4) ; 
for  (x  =  0;  X  <  no.of .vectors;  x++){ 
for  (y  =  0;  y  <  dimension;  y++){ 
fscanf  (in.ptr,  "y,d",ktemp_l) ; 
vector [y]  =  temp.l; 

} 

fscanf  (in.ptr,  "'/,d  \n"  .ftclass)  ; 
if  (class  ==  1){ 

for  (y  =  0;  y  <  dimension;  y++) 

fprintf (out.ptr,"  %d  ",  vector [y] ) ; 
fprintf (out.ptr, "  1  -1  -1  \n"); 

} 

else  if  (class  ”  2){ 

for  (y  *  0;  y  <  dimension;  y++) 

fprintf (out.ptr,"  %d  ", vector [y] ) ; 
fprintf (out.ptr,"  -1  1  -1  \n"); 

} 

else  if  (class  ==  3){ 

for  (y  =  0;  y  <  dimension;  y++) 

fprintf (out.ptr, "  %d  ",vector[y]); 
fprintf (out.ptr,"  -1  -1  1  \n"); 

} 

> 

/*===================== 

Close  Files 

=====================*/ 


f close (in.ptr ' ’ 
f close (out .ptx/ , 

/*======================= 

End  of  Main  Program 
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D.3  How  to  Convert  GTRI  Data  to  Neural  Graphics  Format 


Purpose:  This  program  reads  GTRI  data  files  and  creates  a  Neural 
Graphics  fact  file  for  processing  in  Neural  Graphics.  This 
program  reads  in  data  from  allSO.d  (all  30  classes  of  GTRI’s  data) 
and  then  writes  the  data  in  Neural  Graphics’  format  for  three 
classes. 

Date:  5  June  1992 
Author:  Capt  James  Calvin 

=================================================================*/ 


#include<stdio . h> 
#include<math . h> 
#include<stdlib.h> 

/*================= 

Begin  Main 
=================♦/ 


mainO 

int  X,  y,  dimension=17,  no_of_vectors=5795; 
int  class,  junkl,  junk2,  junkS,  junk4; 

FILE  *in_ptr,  ♦out.ptr; 
int  vector [20]; 
int  temp.l; 

/*============================= 

Array  Declaration 
=============================*/ 


int  features [5795]  [20]  ; 

Open  Files 

in.ptr  =  fopen("c-prcgs/all30 .d' ,"r") ; 
out.ptr  =  fopen("c-progs/ng_3class.dat","w+") ; 
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fscanf  (in_ptr,"7,d  y,d  */,d  ‘/.d  \n'',ftjunkl,tjunk2,ftjunk3,tjunk4) ; 


Read  Data 

for  (x  =  0;  X  <  no.of .vectors;  x++)-( 
for  (y  =  0;  y  <  dimension;  y++){ 
f  scanf  (in.ptr ,  "y,d"  .fttemp.l) ; 
vector [y]  =  temp.l; 

} 

fscanf  (in.ptr ,  "‘/.d  \n" ,  fcclass) ; 


/*============================"== 

Classify  Data 


if  (class  ==  1) 

for  (x  =  0;  X  <  1;  x++)< 

for  (y  =  0;  y  <  dimension;  y++){ 

fprintf (out.ptr,"  yd  ” .vector [y] ) ; 
printfC'I  am  after  class  1  ID  \n"); 

} 

fprintf  (out.ptr,  "1  \n''); 

} 

else  if  (class  ==  2) 

for  (x  =  0;  X  <  1;  x++){ 

for  (y  =  0;  y  <  dimension;  y++)-C 

fprintf (out.ptr,"  Xd  ", vector [y] ) ; 
printfC'I  am  after  class  2  ID  \n"); 

} 

fprintf (out.ptr, "2  \n"); 

} 

else  if  (class  ==  3) 

for  (x  =  0;  X  <  l;x++){ 

for  (y  =  0;  y  <  dimension;  y++){ 

fprintf (out.ptr,"  Xd  " .vector [y] ) ; 


printfC'I  am  after  class  3  ID  \n'') 

} 

f printf  (out _ptr ,  "3  \n'' ) ; 

} 

> 

Close  Files 

fclose(in.ptr) ; 
f close(out_ptr) ; 

End  of  Main  Program 

}  ‘ 
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D.4  Shuffle  Progmm  for  iBrainMaker  Data  Files 

Purpose:  This  program  reads  an  iBrainHaker  fact  file  and 
shuffles  the  facts  (vectors)  so  as  to  create  a  random  fact 
file  for  training  in  the  BrainMaker  simulator.  This  program 
is  limited  to  a  three-class  problem,  but  can  be  extended  to 
more  classes  with  little  modification. 

Date:  6  June  1992 

Author:  Capt  James  Calvin 


#include<stdio . h> 

#include<math . h> 

#include<stdlib . h> 

♦define  loopCA)  forCi  =  0;  i  <  (A);  i++) 
float  temp.l; 
int  iseed; 

FILE  ♦in.ptr,  ♦out.ptr; 

Begin  Main 


mainO 

{ 

int  X,  y,  i=0,  j=0,  idx=0,  count all=0; 
int  vector [20],  used [6000],  order [6000]; 
int  no_of .vectors  =  5795; 
int  array.i; 
int  *array_prt [20] ; 

/*============================ 

Array  Declaration 


int  features [5795] [20]  ; 
iseed  =  2; 
srand (iseed) ; 
countall  =  600; 
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Open  Files 

/♦ — Open  temp  file  (3-class_bm.dat)  and  open  an  output  file 
(3-clas8.bm.dat) — ♦/ 


if((in_ptr  =  fopen("c-progs/3-cla6s_bm.dat'',"r"))==NULL) 

{ 

printfC'I’m  not  able  to  open  the  input  file\n"); 
exit(-l) ; 

} 

if  ( (out_ptr=f open("c-progs/3-class . bm. dat" , "w+") )==NULL) 

printfC'I’m  not  able  to  open  the  output  file\n"); 
exit(-l) ; 

} 


rewind (in.ptr) ; 

Print  a  header  for  the  output  file 


:=♦/ 


fprintf (out _ptr, "Line#  fl  f2  f3  f4  f4  f6  f7  f8  f9 
flO  fll  fl2  fl3  fl4  fl5  fl6  Cl  C2  C3  \n"); 

Read  all  features  into  an  array 


for  (x  =  0;  X  <  count all;  x++) 

fscanf  (in_ptr."y.d  U  7.d  7.d  y.d  U  U  ’/.d  y.d  y.d  %d  U  VA  U 
y,d  %d  7,d  y,d  y,d  y,d",  ^features  [x]  [0]  ,tfeatures  [x]  [1]  , 
^features [x] [2] ,ftf natures  [x] [3] ,&f natures  [x] [4] ,ftf eatures 
[x]  [5]  .tfeatures  [x]  [6]  ,ftfeatures[x]  [7]  .^features  [x]  [8] , 
ftfeatures  [x] [9] .ftfeatures  [x] [10] ,ftf natures  [x] [11] , 
if eatures [x] [12] ,  tf eatures [x] [13] .  &f eatures [x] [14] , 
kfeatures[x] [15] .^features [x] [16] .^features [x] [17] , 
ftf eatures [x] [18] ,4f eatures [x] [19] ) ; 
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Randomize  the  order  of  the  feature  vectors 


j  =  0; 

loop  (countall)-C 
vhile{ 

ifCusedCCidx  =  (int)( (((float) rand ())/RAIfD_MAX)  * 
(countall  -  1)  +  0.5))]==  0) 

used  [idz]  =  1; 
printf  ( '"/.d  \n'' ,  idx)  ; 

order  [j]  =  idx; 

j++; 

break; 

} 

} 

} 


Print  random  vectors  to  a  file 


==♦/ 


y  =  0; 

X  =  0; 

loop  (countall) 

{ 

x=order [y] ; 

f printf  (out_ptr,  "7,d  7,d  7,d  7,d  7.d  7d  7.d  7.d  7.d  7.d  7.d  7.d  7.d 
7td  7d  7td  7.d  7td  7#d  7.d  \n",  features  [x]  [0]  .features  [x]  [1] , 
features [x] [2] .features [x] [3] .features  [x] [4] .  features  [x] [5] . 
features  [x] [6] .features [x] [7] .features  [x] [8] .features  [x] [9] . 
features  [x] [10] .features [x] [11] .features  [x] [12] .features 
[x] [13] .features  [x] [14] .features  [x] [15] .features [x] [16]  . 
features  [x] [17] ,  features [x] [18] .features [x] [19] ) ; 

y++; 

} 
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Close  Files 


/4CSSSSS 

fcloseCin.ptr) ; 
fclose(out_ptr) ; 

/*========================*=== 

End  of  Main  Program 

} 


D.5  Shuffle  Program  for  Neural  Graphics  Data  Files 

Purpose:  This  program  reads  a  Neural  Graphic  fact  file  and 
shuffles  the  facts  (vectors)  so  as  to  create  a  random  fact  file 
for  training  in  the  Neural  Graphic  simulator.  This  program  is 
limited  to  a  three-class  problem,  but  can  be  extended  to  more 
classes  with  little  modification. 

Date:  5  June  1992 


Author:  Capt  James  Calvin 


=============*/ 


#include<stdio . h> 

#include<math . h> 

#include<stdlib . h> 

♦define  loop(A)  for(i  =  0;  i  <  (A);  i++) 
float  temp.l; 
int  iseed; 

FILE  ♦in.ptr,  *out_ptr; 

Begin  Main 


mainO 

int  X,  y,  dimens ion=17,  i=0,  j=0,  idx=0,  count=0,  count all=0; 

int  vector [20],  used [6000],  order [6000]; 

int  no_of .vectors  =  5795; 

int  array. i; 

int  ♦array.prt [20] ; 

/♦============================= 

Array  Declaration 


int  features [5795]  [18]  ; 
iseed  =  2; 
srand(iseed) ; 
countall  =540; 
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count  ■  60; 


Open  Files 

/*— Open  temp  file  (ng_3class.dat)  and  open  an  output  file 

(3-cla88_ng . dat ) — */ 

if((in_ptr  =  fopen("c-progs/ng_3class.dat'',"r"))  ==  NULL){ 
printfC'I’m  not  able  to  open  the  input  file\n"); 
exit(-l); 

} 

if((out_ptr  =  f open(''c-progs/3-clas8 .ng.dat" ,  "w+"))  ==  NULL){ 
printfC'I'm  not  able  to  open  the  output  file\n"); 
exit(-l) ; 

} 

rewind (in_ptr) ; 

Print  a  header  for  the  output  file 
fprintf (out.ptr,"*/,d  '/,d  16  3  \n",  countadl,  count); 


Read  all  features  into  an  array 


for  (x  =  0;  X  <  countall;  x++) 

fscanf  (in_ptr."y.d  U  U  7.d  %d  '/.d  7.d  ’/.d  U  7.d  7.d  7.d  7.d  7.d 
7<d  7d  *IA  7«d",  ftfeatures  [x]  [0]  .defeatures  [x]  [1]  .^features 
[x] [2] , ftfeatures [x] [3] ,  ftfeatures  [x] [4] .ftfeatures  [x]  [5] . 
ftfeatures  [x] [6] .ftfeatures Cx] [7] .ftfeatures  [x] [8] , 
ftfeatures  [x] [9] .ftfeatures [x] [10] .  ftfeatures [x] [11] , 
ftfeatures [x] [12] .  ftfeatures [x] [13] .ftfeatures  [x] [14] . 
ftfeatures [x] [15] .  ftfeatures  [x] [16] .ftfeatures [x] [17] ) ; 
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lUmdomize  the  order  of  the  feature  vectors 

j  =  0; 

loop  (countall)-C 
while(l){ 

if(u8ed[(idx  =  (int)((((float)rand())/RAND_MAX)  ♦ 
(countall  -  1)  +  0.5))]==  0) 

{ 

used  [idx]  =  1; 
printf("*/,d  \n'',idx); 
order  [j]  =  idx; 

break; 

} 

} 

} 

Print  random  vectors  to  a  file 

y  =  0; 

X  =  0; 

loop  (countall) -( 
x=order [y] ; 

fprintf  (out.ptr,  "7.d  7,d  7,d  */,d  7,d  7,d  7.d  7td  7.d  7td  7.d  7.d  7.d 
7.d  7.d  7td  7.d  7.d  \n",  features  [x]  [0]  .features  [x]  [1] , 
feature [x] [2] .features [x] [3] .features  [x] [4] .features [x] [5] . 
features [x] [6] .features [x] [7] .features [x] [8] .features [x] [9] . 
features [x] [10] .features  [x] [11] .features  [x] [12] .features 
[x] [13] .features  [x] [14] .features  [x] [15] .features  [x] [16] . 
features  [x] [17] ) ; 

y++; 

> 
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Close  Files 


/* 


:==*/ 


fcloseCin.ptr) ; 
f close(out_ptr) ; 

/*======================= 

End  of  Main  Program 
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Appendix  E.  C  Programs  to  Normalize  the  Data 


E.l  Introduction 

This  appendix  contains  two  C  programs  for  normalizing  the  iBrainMaker  and 
Neural  Graphics  feature  vectors. 

E.2  Normalization  and  Dynamic  Ranging  for  iBrainMaker  Format 


/*===== 

this  program  was  written  by  don  WILLIS,  GE091D,  AND  IT 
/♦=====  WAS  MODIFIED  BY  JAMES  CALVIN.  GE92D.  THE  PROGRAM 
/»<=»===  STATISTICALLY  NORMALIZES  THE  DATA  SETS  ACCORDING  TO  A 
/4>==sss  mean  and  variance  GAUSSIAN  DISTRTIBUTION 
/♦===== 

/ittass  System  includes  ===========s:====s===s!=====s»s=======s:a:sa*/ 


#include  <stdio.h> 
#include  <math.h> 


/:^==aa==aa=a==a==a==a=aa=a==a==a=aa==aa=====a===a=====aa=a=aaa=a>^/ 

♦define  INPUTS  75  /♦  max  number  of  features  */ 

/*aaa  Main  Routine  ===========================================♦/ 


void  main  (argc.argv) 
int  argc; 
char  *argv  []  ; 

{ 


/*====================================================*/ 

/Dcaaa  local  Variables  ===♦/ 

/*====================================================♦/ 
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FILE  ♦input; 
char  infile [50]; 

FILE  ♦output; 
char  outfile[50]; 
float  trash,  value,  max  -  0; 
char  ♦&; 

float  trash, value; 

double  maxl=0 , max2=0 , max3=0 , maz4=0 , maz5=0 ; 
double  max6=0 ,max7=0 ,max8=0 ,max9=0 ,mazl0s0 ,m2ucll=0 ; 
double  maLXl2=0  ,maxl3=0  ,maxl4=0  ,maxl5=0 , maxi 6=0 ; 
double  fl,f2,f3,f4,f5,f6,f7,f8,f9,fl0,fll,fl2,fl3,fl4; 
double  fl5,fl6; 

float  deviation [INPUTS] ,  average [INPUTS] ; 
int  i,  j,  inputs,  outputs,  i value; 
int  count 1,  count2,  count,  waste,  temp; 
float  valuel,  value2,  value3; 

printf ("Enter  the  number  of  inputs  then  the  number  of 
outputsAn") ; 

scanf("*/.d  y,d",  ^inputs,  ftoutputs); 

printf ("Enter  total  number  of  training/testing  vectors. 
\n"); 

scanf("%d",  tcount); 
fflush(stdin) ; 


/♦====================================================♦/ 

/♦===  did  user  specify  an  input  file  ====♦/ 

/4<=»s==ss====================================:=:====:===*/ 


if  (argc  !=  2) 

printf  ("\n\nUsage  ->  stat-norm  <f ilename>\n\n") ; 
/4c=ss  exit  after  pointing  out  the  error  ===♦/ 
exit  (1000); 

} 
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/♦===  user  did  specify  an  input  file  ===♦/ 

strcpy  (infile,  argv[l]);/*  use  inputted  name  as  base  */ 

/*===  Open  Input  File  ===♦/ 

printf  ("\nOpening  Input  File:  %8\n\n",  infile); 
if  (! (input  =  fopen( infile,  "rb"))){ 

^-intf  ("VnCan't  open  input  file:  XsXnXn",  infile); 
exit  (2000); 

} 

if  (inputs  <  0  1 1  outputs  <  0)-C 

printf  ("One  of  the  header  inputs  is  negative\n\n") ; 
exit  (3000) ; 

} 

printf  ("There  are  */,d  training/testing  vectorsXn",  count); 
printf  ("There  are  */.d  inputsXn",  inputs); 
printf  ("There  are  7,d  outputsXnXn",  outputs); 


/♦================="==============*==============””=♦/ 

/*===  initialize  things  ===*/ 

/♦=========.=======================:=.=================♦/ 

for  (i  =  0;  i  <  inputs;  i++) 

average [i]  =  deviation [i]  =  0.0; 

/*====================================================♦/ 

loop  until  all  data  has  been  read  in  ===*/ 

/♦===========^_=======================================*/ 


printf  ("Reading  the  DataXnXu"); 

fscanf  (input, "7,s  ’/.s  7.s  */,s  ‘/.s  ‘/.s  7,s  7.s  7,s  7.8  7.s  7.s  7.s  7.s  7.s 
7.S  7tS  7.S  7s  7.S  Xn",  &&,  &s,  ks,  ks,  ka,  ks,  ks,  ke,  ks,  ks, 
fts,  ks,  ks,  ks,  ks,  ks,  ks,  ks,  ks,  ks); 

fflush(8tdin) ; 

for  (i  =  0;  i  <  count;  i++){ 
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fscanf  (input.  ''%d  ",  ittrash);/*  read  line  counter  */ 
for  (j  =  0;  j  <  inputs;  j++){ 
fscanf  (input.  "5if  ftvalue); 
average  [j]  •♦■=  value; 

} 

fscanf  (input,  "Xd  %d  y,d  \n",  iitrash,  fctrash,  fctrash); 

} 

f close  (input); 

/*===  calculate  the  averages  ===♦/ 

printf  ("Calculating  Averages\n\n") ; 
for  (i  =  0;  i  <  inputs;  i++) 
average [i]  /=  (float)count; 


/*===  Re-open  the  input  file  ===♦/ 

printf  ("Re-Opening  Input  File:  %8\n\n",  infile); 
if  ({(input  =  fopen(infile,  "rb"))){ 

printf  ("XnCan’t  re-open  input  file:  %s\n\n",  infile); 
exit  (2000); 

} 

/*sss  loop  until  all  data  has  been  read  in  ===♦/ 

fscanf  (input.  "Xs  y.s  */.s  %s  %s  %s  %8  %8  y,s  Xs  '/.s  '/.s  Xs  '/.s 
Xs  Xs  Xs  Xs  Xs  \n",  ks,  ks,  ks,  ke,  ka,  ks,  ke,  ka,  ka,  ka, 
ka ,ka ,ka ,ka ,ka ,ka ,ke .ka ,ka .ka) ; 

ff lush(stdin) ; 

printf  ("Reading  the  Data\n\n"); 
for  (i  »  0;  i  <  count;  i”M-)>C 

fscanf  (input,  "Xd  ",  ttrash); 
for  (j  *  0;  j  <  inputs;  j++)-C 

fscanf  (input,  "Xf  ",  ftvalue); 
value  -«  aver age [j]; 
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value  *=  value; 

deviation [j]  +=  value; 

} 

fscanf  (input,  '"/.d  7,d  *>Cd  \n'',  ttrash,  fttrash,  fctrash); 

} 

f close  (input) ; 


/*====================================================*/ 

/*===  calculate  the  standard  deviation  ===♦/ 


printf  ("Calculating  Standard  Deviations\n\n") ; 
for  (i  =  0;  i  <  inputs;  i++){ 
deviation [i]  /=  count  -  1; 

deviationCi]  =  (float)sqrt((double)deviation[i] ) ; 


/*===================================================*/ 

/i4is==  make  output-file  name  ===♦/ 

sprintf  (outfile,  "!Cs.sn",  argvCl]); 

/iti===  Open  Output  File  ==♦/ 

printf  ("Opening  Output  File:  )is\n\n",  outfile); 


if  (!  (output  =  f  open  (outfile,  "wb")))-C 

printf  ("XnCan't  open  output  file:  %s\n\n",  outfile); 
exit  (2000); 


/♦===  Re-open  the  input  file 


printf  ("Re-Opening  Input  File  (last  time)  :y.s\n\n",  infile)  ; 
if  (Kinput  *  fopen(infile,  "rb")))^ 

printf  ("\nCan’t  re-open  input  file:  7,s\n\n",  infile); 
exit  (2000); 
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/♦=«*  read  data  in,  modify  it,  save  it  back  out  ==♦/ 

printf  (“Reading,  Modifying  and  Re-Saving  the  Data\n\n“); 
fscanf  (input, "*/.s  */,s  */.s  y.s  */.s  */.s  */,s  %8  Xs  '/.s  7.s  Xs  %s  '/.s  7,8 
%s  Xs  7s  78  78  \n“,  ta,  ka,  ka,  ka,  ka,  ks,  ks,  ks,  ks,  ks, 
ks,  ks,  ks,  ks,  ks,  ks,  ks,  ks,  ks,  ks); 

fflush(stdin) ; 

for  (i  =  0;  i  <  count;  i++)-C 

fscanf  (input, "7d", kivalue) ; 
f printf  (output, "7d",ivaiue); 

for  (j  =  0;  j  <  inputs;  j++){ 
fscanf (input , "7f “ i kvalue) ; 
value  -=  average [j]; 
value  /=  deviation Cj] ; 
fprintf (output, "7f", value) ; 

} 

fscanf (input, “7d  7d  7d  \n“,kivaluel,kivalue2,kivalue3) ; 
fprintf (output, “7d  7d  7d  \n ",i value 1, ivalue2, ivalueS ) ; 

} 

f flush (output) ; 


/♦=®=Normalize  Data  Between  -1  and  i=======s*===*s==*/ 

/»sss  nake  output-file  name  «==♦/ 

sprintf  (infile,  “78. sn",  argvCl]); 
sprintf  (outfile,  “7s. n“,  argv[l]); 

/4CSSS  Open  Output  File  «*=♦/ 


printf  ("Opening  Output  File:  78\n\n",  outfile); 
if  ('(output  ■  f  open  (outfile,  "wb")))-C 
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printf  ("\nCan*t  open  output  file:  %8\n\n",  outfile); 
exit  (2000); 


/*===  Re-open  the  input  file 

printf  ("Re-Opening  Input  File  (last  time) :%8\n\n", infile); 
if  ({(input  =  fopen (infile,  "rb”))){ 

printf  ("\nCan*t  re-open  input  file:  */,s\n\n",  infile); 
exit  (2000); 


/♦================*============================ 

/desss  read  data  in,  modify  it,  save  it  back  out 


printf  ("Reading  the  Data\n\n"); 
for  (i  *  0;  i  <  count;  !++)•( 

fscanf  (input,  "'/.d  ",  tivalue); 

fscanf  (input,  "’/.If  ’/.If  7.1f  '/.If  %lf  7.1f  */.lf  7.1f  %lf  %lf 
•/.If  %lf  %li  '/.If  ’/.If  ’/.If",  ftfl,tf2,kf3.Jkf4,kf5,kf6, 
tf7,«8,tf9,ftfl0,4fll,ftfl2,ftfl3,kfl4,fcfl5,kfl6); 
if  (fabs(fl)  >  maxi) 
maxi  ~  fabs(fl); 
if  (fabs(f2)  >  max2) 
max2  =  fabs(f2); 
if  (  fabs(f3)  >  max3) 
max3  ~  fabs(f3); 
if  (fab8(f4)  >  max4) 
max4  =  fabs(f4); 
if  (fabs(f5)  >  max5) 
maxS  s  fabs(f5); 
if  (fab8(f6)  >  max6) 
iiiax6  -  fabs(f6); 
if  (fab8(f7)  >  max7) 
max7  -  fabs(f7); 
if  (fabs(f8)  >  max8) 
max8  ■  fabs(f8); 
if  (fabs(f9)  >  max9) 


=======♦/ 

«=♦/ 

=======*/ 
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max9  *  fabsCfQ); 
if  (fabs(flO)  >  maxlQ) 
mazlO  s  fabsCflO); 
if  (fabsCfll)  >  maxll) 
maxll  -  fabsCfll); 
if  (fab8(fl2)  >  mazl2) 
mazl2  -  fabs(fl2); 
if  (fabsCflS)  >  maxis) 
maxis  -  fabs(flS): 
if  (fab8(fl4)  >  maxl4) 
m2ucl4  s  fab8(fl4); 
if  (fabsCflS)  >  maxlS) 
maxis  =  fabsCflS); 
if  CfabsCfie)  >  maxl6) 
maxis  =  fabsCflS); 

fscanf  Cinput,"*/f  *ld  7.f  \n",  lcvaluel,JEvalue2, 
lvalues) ; 

} 

fseekCinput,0.O) ; 

/*sss  read  data  in,  modify  it,  save  it  back  out  ===*/ 

printf  C'Reading,  Modifying  and  Re-Saving  the  Data\n\n"); 

>K  Printing  a  header  for  a  S-class  problem 

****4<****«****««*4>«««**>l»l<*«4i***«***«««)tc*4c*4c*)|t*********«****)tc)(t:tc*/ 

f printf Coutput, "Line#  fl  f2  fS  f4  fS  fS  f7  f8  f9 

fio  fll  fl2  fis  fl4  fis  fis  Cl  C2  CS  \n"); 

for  Ci  *  0;  i  <  count;  i++)< 

fscanf  Cinput,  "%d  ",  kivalue); 

f printf  Coutput,  "Xd  ",  ivalue); 

fscanf  Cinput,  "%lf  %lf  7.1f  %lf  Xlf  %lf  Xlf  Xlf  7.1f 

Xlf  Xlf  Xlf  Xlf  Xlf  Xlf  Xlf,  tfl,«2.«S,«4,kfS, 

JtfS,  «7,  kf8,  «9,  «10,  tfll,  ftfl2,  «13,  «14, 
kflS,  tflS); 
fl  /s  maxi; 
if  Cfl  >«  0.0) 
fl  +*  O.OOOS; 
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else 

fl  -+  0.0005; 
f2  /«  Daz2: 
if  (f2  >*  0,0) 
f2  +=  0,0005; 
else 

f2  -+  0.0005; 
f3  /«  maxS; 
if  (f3  >*  0.0) 
f3  +=  0.0005; 
else 

f3  -+  0.0005; 
f4  /=  max4; 
if  (f4  >*  0.0) 
f4  +=  0.0005; 
else 

f4  -+  0.0005; 
fS  I-  max5; 
if  (f5  >=  0.0) 
f5  +=  0.0005; 
else 

f5  -+  0.0005; 
f6  /*  max6; 
if  (f6  >«  0.0) 
f6  +=  0.0005; 
else 

f6  -+  0.0005; 
f7  /=  maxT; 
if  (f7  >=  0.0) 
f7  +=  0.0005; 
else 

f7  -+  0.0005; 
f8  /=  maz8; 
if  (f8  >=0.0) 
f8  +»  0,0005; 
else 

f8  -+  0.0005; 
f9  /«  max9; 
if  (f9  >«  0.0) 
f9  +*  0.0005; 
else 

f9  -+  0.0005; 
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flO  /«  maxlO; 
if  (flO  >»  0.0) 
flO  +»  0.0005; 
else 

flO  -+  0.0005; 
fll  /*  maxll; 
if  (fll  >=  0.0) 
fll  +«  0.0005; 
else 

fll  -+  0.0005; 
fl2  /s  mazl2; 
if  (fl2  >*  0.0) 
fl2  +=  0.0005; 
else 

fl2  -+  0.0005; 
fl3  /s  maxl3; 
if  (fl3  >=  0.0) 
fl3  +=  0.0005; 
else 

fl3  -+  0.0005; 
fl4  /s  mazl4; 
if  (fl4  >=  0.0) 
fl4  +*  0.0005; 
else 

fl4  -+  0.0005; 
fl5  /s  maxl5; 
if  (fl5  >*  0.0) 
fl5  +=  0.0005; 
else 

fl5  -+  0.0005; 
fl6  I-  mazlO; 
if  (fl6  >s  0.0) 
fl6  +=  0.0005; 
else 

fl6  -+  0.0005; 

fprintf  (output,  •'  */.3i;  %.31f  %.31f  %.31f  X.31f 
%.31f  %.31f  %.31f  X.31f  X.31f  %.31f  X.31f  %.31f 
X.31f  X.31f  X.31f  fl,f2,f3,f4,f5  ,f6,f7,f8  ,f9, 

flO,fll,fl2,fl3,fl4,flS,fl6) ; 
fscanf  (input,  "Xf  Xf  Xf  \n",  tvaluel,  fcvalue2, 
kvalue3) ; 
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fprintf  (output.  “51. If  %.lf  it. If  \n",  value l,value2. 
values) ; 

} 

f close  (input); 
f close  (output) ; 

/*sss  we* re  done  ===♦/ 

printf  ( "Finished . \n\n" ) ; 

} 


E.8  Normalization  and  Dynamic  Ranging  for  Neural  Graphics  Format 


/*===== 

/♦=====  this  program  was  originally  written  by  don  WILLIS,  GE091D, 
/♦=====  and  it  was  modified  by  JAMES  CALVIN,  GE92D.  THE  PROGRAM 
/♦=====  STATISTICALLY  NORMALIZES  THE  DATA  SETS  ACCORDING  TO  A 
/♦=====  mean  and  variance  GAUSSIAN  DISTRIBUTION,  AND  THEN  IT  RE- 
/*=====  duces  the  dynamic  range  to  -1.0  TO  1.0. 

/♦===== 

/*===  System  includes  =========================================== 

#include  <stdio.h> 

#include  <math.h> 

/*======================”==“=♦/ 

/*===  Defines  ===============♦/ 

/«=============s===============4c/ 

tdefine  INPUTS  75;  /*  majc.  number  of  features  */ 

/*===  Main  Routine  =========♦/ 

void  main  (argc,argv) 
int  argc; 
char  ♦argv  []  ; 

{ 

/*===  local  variables 

FILE  ♦fopenO; 

FILE  ♦input,  ♦fopenO; 
char  infile  [50],* 

FILE  ♦output; 
char  outfile[50]; 
float  trash, value; 


double  aaxlsO , Bax2«0 , maxS^O , max4«0 , maxS^O ; 
double  max6«0 , max7«0 , max8«0 . iiiax9«0 . maxl0«0 ,  maxi  1^0 ; 
double  maxl2>0 ,maxl3«0 .maxl4s0 ,maxl5«0 ,maxl6«0 ; 
double  fl,f2.f3.f4,f5.f6,f7,f8,f9,fl0.fll,fl2,fl3,fl4; 
double  fl5,fl6; 

float  deviation [INPUTS] ,  average [INPUTS] ; 

int  i,  j.  inputs,  outputs,  ivalue; 

int  count 1,  count2,  count,  waste,  temp; 

/41SSS  did  user  specify  an  input  file  ««==♦/ 

if  (argc  !«  2) 

printf  (''\n\nUsage  ->  stat-norm  <filename>\n\n'') : 
/*sss  exit  after  pointing  out  the  error  «=*♦/ 
exit  (1000); 

} 

/♦===  user  did  specify  an  input  file  =*£♦/ 

/^aagaaaagRBaagaaaaagsassaaaaaaaaaagaaaaaaaaaaasaagaaasv/ 


strcpy  (infile,  argv[l]):/*  use  inputted  name  as  base  */ 


printf  C'XnOpening  Input  File;  %s\n\n",  infile); 
if  (! (input  »  fopen(infile,  "rb"))) 

printf  C'NnCan’t  open  input  file:  %s\n\n",  infile); 
exit  (2000); 

} 

*/ 
♦/ 
♦/ 

fscanf (input, "%d  %d  %d  %d\n",itcountl,kcount2,tinputs. 


/4iaaa  read  the  header  information 

/*aaaaaaaaa«aaaaaaaaaaaaaaaaaaa.» 
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^outputs) ; 

if (count 1  <  0  i I  count2  <  0  1 1  inputs  <  0  1 1  outputs  <  0) 

{ 

printf  ("One  of  the  header  inputs  is  negative\n\n") ; 
exit  (3000); 

} 

printf  ("There  are  Xd  training  vectorsNn",  countl); 
printf  ("There  are  %d  test  vectors\n",  count2): 
printf  ("There  are  Xd  inputs\n".  inputs); 
printf  ("There  are  Xd  output8\n\n",  outputs); 
count  >  countl  count2; 

/*sss  initialize  things 

/ *SBSSSSSSXSSSSSSS=SSSS 


for  (i  =  0;  i  <  inputs;  i++) 

average [i]  =  deviation [i]  =  0.0; 

} 

/4i«Bss««««ssa;ssBSSBsssssssssssssssaEssssssssssa»sssx»ss4i/ 

/«»«  loop  until  all  data  has  been  read  in  «=♦/ 


printf  ("Reading  the  Data\n\n"); 
for  (i  “  0;  i  <  count;  i-M-) 

{ 

fscanf (input, "%d  ",fctrash);/e  read  line  counter  */ 
for  (j  ®  0;  j  <  inputs;  j++) 

fscanf (input, "%f  ", Rvalue ) ;/e  read  float  values  */ 
average [j]  +«  value; 

} 

fscanf  (input,  "Xd  \n",  Rivalue); 

} 

f close  (input); 


/** 

/*ggg  calculate  the  averages 
/*- 


*»*/ 

=g*/ 
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printf  ("Calculating  Averages\n\n") ; 
for  (i  =  0;  i  <  inputs;  i++) 

average [i]  /=  (float) count ; 

} 

/♦===  Re-open  the  input  file  ===♦/ 

printf  ("Re-Opening  Input  File:  X8\n\n",  infile); 
if  (! (input  =  fopen(infile,  "rb"))) 

{ 

printf  ("\nCan’t  re-open  input  file:  */,s\n\n",  infile); 
exit  (2000); 

} 

/*===  throw  away  the  header  information  this  time  ===♦/ 

fscanf (input, "%d  */«<i  *IA  )ld\n",lEwaste,A:wa8te,ftwaste,Awaste) ; 

/4c=ss  loop  until  all  data  has  been  read  in  ===♦/ 

printf  ("Reading  the  Data\n\n"); 
for  (i  =  0;  i  <  count;  i++) 

■C 

fscanf  (input,  "5(d  ",  fttrash);/*  read  line  counter  */ 
for  (j  =0;  j  <  inputs;  j++) 

{ 

fscanf  (input, "%f ", tvalue) ;/♦  read  values  ♦/ 

value  -*  average [j];  /*8ubtract  off  the  average*/ 

value  *=  value;  /*  square  the  result  */ 

deviation  [j]  value ; /*hsLng  onto  it  until  edl  done*/ 

} 

fscanf  (input,  "*/,d  \n",  Aivalue); 

} 

f close  (input); 
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/itcsss  calculate  the  standard  deviation  ss=4c/ 

printf  (“Calculating  Standard  DeviationsXnXn") ; 
for  (i  =  0;  i  <  inputs;  i++) 
deviation [i]  /=  count  -  1; 

deviation[i]  =  (float)sqr^((double)deviation[i] ) ; 

} 


/*===  make  output-file  name  ===♦/ 

sprintf  (outfile,  '"yls.sn",  argv[l]); 

/itcssa  Open  Output  File  ===*/ 


printf  (“Opening  Output  File:  %s\n\n”,  outfile); 
if  (! (output  *  fopen(outfile,  “wb"))) 

{ 

printf  (“XnCan't  open  output  file:  7,s\n\n",  outfile); 
exit  (2000); 

} 


/♦=**=***==*===============—==»===—=======♦/ 

/itcass  Re-open  the  input  file  ===♦/ 


printf  ("Re-Opening  Input  File  (last  time)  :7,B\n\n",  infile ) ; 
if  (! (input  B  fopen(infile,  “rb"))) 

printf  (“XnCan't  re-open  input  file:  %s\n\n",  infile); 
exit  (2000); 

} 


/*- 

/♦. 


read  and  save  header 
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fscanf (input, "%d  %d  %d  %d\n“,fccountl,itcount2,fcinput8, 
^outputs) ; 

fprintf (output, "%d  %d  *>Cd  5id\n'', count l,count2, inputs, 
outputs) ; 

/*===  read  data  in,  modify  it,  save  it  back  out  ===♦/ 

printf ("Reading,  Modifying  and  Re-Saving  the  Data\n\n"); 
for  (i  =  0;  i  <  count;  i++) 

fscanf (input , "*/,d  ",tivalue) ;/*read  line  counter*/ 
f  printf  (output,  "y,d  ",  i  value  );/*save  line  counter*/ 
for  (j  =0;  j  <  inputs;  j++) 

fscanf  (input, "*/,f",kvalue)  ;/*  read  float  value  */ 
value  -=  average [j] ;/*  modify  the  value  */ 
value  /=  deviation Cj] ; 

fprintf (output, "*/.f", value) ;/*8ave  modified  value*/ 

} 

fscanf  (input,  "%d  \n",  kivalue); 

fprintf  (output,  "%d  \n",  ivalue); 

} 

ff lush(output) ; 

/♦==Normalize  Data  Between  -1  and  ics======ssss=*/ 

*/ 
*/ 


sprintf  (infile,  "Xs.sn",  argv[l]}; 
sprintf  (outfile,  "%s.n",  argvCl]); 


/esss  inake  output-file  name 
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printf  ("Opening  Output  File:  */i8\n\n",  outfile); 
if  (! (output  *  fopen(outf ile,  "wb"))) 

{ 

printf  ("XnCan’t  open  output  file:  y,s\n\n",  outfile); 
e:pit  (2000); 

} 

/*===  Re-open  the  input  file  ===♦/ 

printf  ("Re-Opening  Input  File  (last  time)  :*/,s\n\n",  inf  ile) ; 
if  (! (input  =  fopen(inf ile,  "rb"))) 

{ 

printf  ("\nCan*t  re-open  input  file:  y,s\n\n",  infile); 
exit  (2000); 

} 

/ ♦»==========»================= 

/♦===  read  the  header 

fscanf (input, "%d  %d  Xd  Xd\n",ftcounti,ftcount2,hinput8, 
^outputs) ; 


/)tcs»s  read  data  in,  modify  it,  save  it  back  out  ===♦/ 

printf  ("Reading,  Modifying  and  Re-Saving  the  Data\n\n"); 
for  (i  =  0;  i  <  count;  i++)-C 

fscanf  (input,  "Xd  ",  fci value);  /♦  read  line  counter  ♦/ 
fscanf  (input,  "Xlf  Xlf  Xlf  Xlf  Xlf  Xlf  Xlf  Xlf  Xlf  Xlf 
Xlf  Xlf  Xlf  Xlf  Xlf  Xlf",  «l,«2,«3,«4,tf5,kf6. 
kf7,ftf8,kf9,Jtfl0,kfll,kfl2,kfl3,kfl4,kfl5,kfl6); 
if  (fab8(fl)  >  maxi) 
maxi  s  fabs(fl); 
if  (fabs(f2)  >  max2) 
maz2  s  fabs(f2); 
if  (fab8(f3)  >  max3) 
max3  *  fabs(f3); 
if  (fabs(f4)  >  max4) 
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iiiaz4  s  fab8(f4); 
if  (fabsCfS)  >  max5) 
mazS  «  fabs(fS): 
if  (fabsCfe)  >  maze) 
maz6  -  fabs(f6); 
if  (fabs(f7)  >  maz7) 
maz7  =  fab8(f7); 
if  (fab8(f8)  >  maz8) 

]iiaz8  -  fab8(f8): 
if  (fab8(f9)  >  maz9) 
maz9  =  fab8(f9); 
if  (fab8(flO)  >  mazlO) 
mazlO  =  fab8(flO): 
if  (fab8(fll)  >  mazll) 
mazll  =  fab8(fll): 
if  (fab8(fl2)  >  mazl2) 
mazl2  =  fab8(fl2); 
if  (fab8(fl3)  >  mazlS) 
mazlS  s  fab8(fl3): 
if  (fab8(fl4)  >  mazl4) 

Bazl4  =  fab8(fl4); 
if  (fab8(fl5)  >  nazlS) 
aazlS  s  fab8(fl5); 
if  (fab8(fl6)  >  mazl6) 
mazlS  =  fabaCfie); 
facanf  (input, ''%d  \n",  tivalue); 
} 

f8eek(input,0,0) ; 


facanf (input, ''%d  %d  %d  Xd\n",lecountl,tcount2,tinput8, 
&output8) ; 

fprintf ( output, "%d  %d  %d  Xd\n'', count l,count2,input8, 
output 8) ; 


/*» 

/*> 

/♦=« 


read  data  in,  modify  it,  nave  it  back  out 


«♦/ 

'*/ 

»*/ 
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printf  ("Reading,  Modifying  and  Re-Saving  the  Data\n\n"): 


for  (i  «  0;  i  <  count;  i++){ 

fscanf  (input,  ")^d  ",  Ifcivalue) ; />^read  line  counter*/ 
f printf  (output,  "i(d  ",  ivalue) ; /*save  line  counter*/ 
fscanf  (input,  "‘/.If  7.1f  */.lf  %lf  %lf  */lf  Xlf  %lf  7.1f 
Uf  Xlf  7.1f  XU  XU  7.1f  nf".  kfl,4f2.Rf3,l:f4,kf5,«6, 
kfl.  If 8,  tf9,  tflO,  kill,  Itfl2,  tfl3,  kfl4,  tfl5,  kfl6): 
fl  /=  maxi; 
if  (fl  >»  0.0) 
fl  +=  0.0005; 
else 

fl  -+  0.0005; 
f2  /=  max2; 
if  (f2  >*  0.0) 
f2  +=  0.0005; 
else 

f2  0.0005; 
f3  /«  maxi; 
if  (f3  >»  0.0) 
f3  +s  0.0005; 
else 

f3  -+  0.0005; 
f4  /s  maxi; 
if  (f4  >«  0.0) 
f4  +*  0.0005; 
else 

f4  -+  0.0005; 
f5  /-  maxi; 
if  (f5  >*  0.0) 
f5  +*  0.0005; 
else 

f5  -+  0.0005; 
f6  /s  maxi; 
if  (f6  >«  0,0) 
f6  +=  0.0005; 
else 

f6  -+  0.0005; 
f7  /*  maxi; 
if  (f7  >•  0.0) 
fl  +«  0.0005; 
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else 

f7  -+  0.0005; 
f8  /=  maxi; 
if  (f8  >»  0.0) 
f8  +=  0.0005; 
else 

f8  -+  0.0005; 
f9  /a  maxi; 
if  (f9  >=  0.0) 
f9  +a  0.0005; 
else 

f9  -+  0.0005; 
flO  /=  maxi; 
if  (flO  >=  0.0) 
flO  +=  0.0005; 
else 

flO  -+  0.0005; 
fll  /=  maxi; 
if  (fll  >=  0.0) 
fll  +=  0.0005; 
else 

fll  -+  0.0005; 
fl2  /«  maxi; 
if  (fl2  >s  0.0) 
fl2  +=  0.0005; 
else 

fl2  -+  0.0005; 
fl3  /s  maxi; 
if  (fl3  >=  0.0) 
fl3  0.0005; 

else 

fl3  -+  0.0005; 
fl4  /a  maxi; 
if  (fl4  >=  0.0) 
fl4  0.0005; 

else 

fl4  -+  0.0005; 
fl5  /a  maxi; 
if  (fl5  >»  0.0) 
fl5  +«  0.0005; 

else 

fl5  -+  0.0005; 
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fl6  /«  maxi; 
if  (fl6  >»  0.0) 
fl6  ♦=  0.0005; 
else 

fl6  -+  0.0005; 

fprintf  (output.  "  */..31f  l.Zlf  %.31f  %.31f  %.31f 
%.31f  %.3U  •/.31f  •/.31f  •/..31f  %.31f  X.31f  )i.31f 
•iC.31f  X.31f  %.31f  •'.fl.f2.f3.f4.f5.f6,  f7,f8,f9, 
fl0,fll,fl2.fl3.fl4.fl5.fl6); 
fscanf  (input.  "  y,d  \n'',  Aivalue); 
fprintf  (output.  "  */,d  \n''.  ivalue); 

} 

f close  (input); 
f close  (output); 

/♦*==  we’re  done  ===♦/ 


printf  ("Finished. \n\n"); 

} 

/sssssssssssssssasvsssssssss 
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Appendix  F.  Neural  Graphics  Module  Modification  for  ETANN  Chip 

Simulation 


F.  1  Introduction 

This  appendix  contains  the  backpropagation  C  module,  jcbackprop.c,  for  Neu¬ 
ral  Graphics  that  was  modified  to  emulate  the  sigmoidal  transfer  function  and  to  limit 
weight  values  to  within  ETANN’s  dynamic  range.  The  Neural  Graphics  executable 
file  name  was  changed  from  net  to  jcnet.  Hence,  jcbackprop.c  along  with  modified 
files  saver. c,  general. c,  and  makeinput.c  axe  all  the  files  affected  by  the  ETANN  em¬ 
ulation.  General,  c  and  saver,  c  modifications  were  presented  in  Appendix  B.  Below 
are  the  modifications  to  makeinput.c  and  backprop.c  (note:  backprop.c  was  replaced 
with  jcbackprop.c  which  is  included  below). 


Replace  "case  CLASS"  in  makeinput.c  with  the  following  changes: 

/  *  *  ]|t  *  itr  4[  itciti  Hi  Hi  4c  4i  4r  ^  4i  itr  4i  i|r  4e  #  41 4i  *  *  *  «  *  *  *  *  *  Hi  4c  *  4t  *  *  4i  *  *  *  *  *  *  * 


case  CLASS  :  loopi (output) {  if(i~ctype) 

dof t [parameters . output- 1-i] 


else 

dof t [parameters. output-l-i]  «  -0.94;} 


0.89; 


4c4i4i4i4t4i4i4i4i4i4i4i4c4i4i4i4i4i4i4i4i4i4i4i4i4!4c4i4i4i4i4i4i4c4i4c4i4i4i4i4i*4i4i4>4i4i4i4i4i*4i4c4i4>4i4c4>*4i4c4i4i4i4i4i4c4c/ 

Replace  backprop.c  with  the  following  module  and  either  rename  the 
file  as  jcbackprop.c  (requires  changing  the  makefile:  renaming 
executable  command  and  adding  jcbackprop.c)  or  leave  alone  and  note 
that  the  simulator  is  now  operating  only  as  an  ETANN  simulator. 
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♦ 

*  DATE:  1  Aug  1992 

*  VERSION:  3 

* 

*  NAME:  Module  for  Backprop 

«  MODULE  NUMBER:  1.6 

*  DESCRIPTION:  Using  output,  adjusts  weights,  k  reduce  error. 

*  ALGORITHM:  Modified  Werbos  Multilayer  Perceptron  Back 

*  propagation. 

*  PASSED  VARIABLES:  None 

*  RETURNS:  None 

*  GLOBAL  VARIABLES  USED:  Weight  Vectors 

*  GLOBAL  VARIABLES  CHANGED:  Weight  Vectors 

«  FILES  READ:  None 

*  FILES  WRITTEN:  None 

*  HARDWARE  INPUT:  None 

*  HARDWARE  OUTPUT:  None 

*  MODULES  CALLED:  None 

*  CALLING  MODULES:  Main  Loop 

* 

*  AUTHOR:  Gregory  L.  Tarr 

*  HISTORY:  Modified  by  Capt  Jaises  Calvin  for  ETANN  chip 

*  Siaulation 

* 

finclttde  "definitions.h" 
tinclude  <math.h> 

extern  nnet  *net; 

extern  setup  parameters; 

extern  examplars,  count; 

extern  void  TRAIN.NULLO ; 

float  calcyO  ,calcm()  ,f ixy()  ,calcy.linear() ; 

float  calcy.nomaskC) , calcy.linear.nomaskO ; 

float  hardness  »  i.O; 

void  TRAIN_LOWER.UyER()  ,TRAIN.OUTPUT_LAYER() ; 
init .train_BACKPROP ( ) < 
int  i; 

init.net (parameters) ; 
loop! (parameters . layers ) { 
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net->layerl [i] ->updater  =  TRAIN.LOWER.LAYER; 
net->layerl[i]->etaCO]  =0.3; 
net->layerl[i]->eta[l]  =0.7; 

} 

net->layerl [parameters . layers- 1] ->updater=TRAIN_OUTPUT_LAYER: 
loop! (parameters . layers) 
net->layerl[i]->propagate  =  calcy; 

} 

void  TRAIM.OUTPUT.LAYER  (l.on.l-up) 
layer  ♦l_on,*l_up;{ 
int  i.j.k. lower, upper; 
float  net a. momentum; 
lover  =  l_on->size_input; 
upper  =  l_on->si2e_output ; 

neta  =  l_on->eta [0]/ (float) l_on->si2e_input; 
momentum  =  l_on->eta[l] ; 

dely (l_on->output , net->dof t , l_on->delta, upper ) ; 

/**Bias  weights  constrained  between  -15  and  15  for  6  ETAKN  biases**/ 
loop j (upper)  < 

if  (fabs( (double) l_on->thetaCj])  <=  15.0) 
l_on->theta[j]  =  l.on->theta[j]  +  neta  *  l_on->delta[j]  ; 

if  (fabs( (double) l.on->theta[j])  >  15.0){ 
if  (l_on->theta[j]  >=  0.0)  l.on->theta[j]  =  15.0; 
if  (l_on->theta[j]  <  0.0)  l.on->theta[j]  =  -15.0; 

} 

loopi ( l_on->  s ize_ input ) { 

/***Weights  constrained  between  -2.5  and  2.5  for  ETANN  chip***/ 

if  (fabs ( (double) l_on->weights [j] [i])  <=  2.50) 
l_on->weights[j]  [i]  *•= 

neta  *  l_on->delta[j]  *  l.on->input [i]  *  l_on->mask[i] ; 
if  (fabs ( (double) l.on->veights [j] [i])  >  2.50){ 

if  (l_on->weights[j] [i]  >=  0.0)  l_on->weight8[j] [i]  *  2.50; 
if  (l_on->weights[j] [i]  <  0.0)  l_on->weights [j] [i]  *  -2.50; 

} 

} 

} 

> 

void  TRAIN_L0WER_LAYER(l.on,l.up) 
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layer  ♦l_on,’*‘l_up;-C 
int  i, j ,k, lower, upper; 
float  net a, momentum; 
lover  *  l_on->size_input; 
upper  =  l_on->si2e_output; 
neta  =  l_on->eta[0]/l_on->si2e_input ; 
momentum  =  l_on“>eta[l] ; 
delx(l_on->output , l_up->delta, l_on->delta, 
l_up->weight8 , l_up->si2e_output .upper) ; 

/**Bias  weights  constrained  between  '15  zmd  15  for  6  ETANN  biases*’*'/ 

loopj (upper)  { 

if  (fabs( (double) l_on->thetaCj])  <=  15.0) 
l_on->theta[j]  =  l_on->theta[j]  +  neta  l_on->delta[j]  ; 

if  (fabs((double)l_on->theta[j])  >  15.0){ 
if  (l_on->theta[j]  >=  0.0)  l_on->theta[j]  =  15.0; 
if  (l_on“>theta[j]  <  0.0)  l_on->theta[j3  =  -15.0; 

} 

loopi(l_on->si2e_input)  { 

/’►’•'♦♦’•'Weights  constrained  between  -2.5  and  2.5  for  ETANN  chip  ♦♦♦♦♦/ 

if  (fabs ( (double) l_on->weight8 Cj] [i])  <=  2.50) 
l_on->weights [j] [i]  +=  neta  ♦  l_on->delta[j3  * 
l_on->input [i]  ♦  l_on->mask [i] ; 

if  (fabs ( (double) l_on->weights Cj] [i])  >  2.50){ 
if  (l_on->weightsCj] [i]  >=  0.0)  l_on->weight8[j] [i]  =  2.50; 
if  (l_on->weightsCj] [i]  <  0.0)  l_on->weight8 [j]  [i]  =  -2.50; 

} 

} 

} 

} 

float  calcy(j ,l_on) 
int  j; 

layer  ♦l_on;< 
int  i', k, number; 
float  y; 

return  f ixy (calcy.linear ( j , l_on) ,1.0);} 
float  calcy_linear(j ,l_on) 
int  j; 

layer  ♦l.on;{ 
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int  i; 
float  y; 
y  »  0.0; 

loopi(l_oii->8ize_input){  if (l_oii->mask[i]  “  0.0)  continue; 
y  +=i_on->input [i]  ♦  l.on->weights[j] [i] ; 

} 

y  +»  l_on->theta[j] ; 
return  y; 

} 

float  calcy.nomaskCj ,l.on) 
int  j; 

layer  *l_on;{ 
int  i.k, number; 
float  y; 

y  =  calcy.linear.nomaskCj ,l.on) ; 
return  f ixy (calcy_linear( j , l_on) ,1.0); 

} 

float  calcy.linear.nomaskCj ,l_on) 
int  j; 

layer  ♦l_on;{ 
int  i,k, number; 
float  y; 
y  »  0.0; 

loopi(l.on->si2e_input)  { 

y  *  y  +  l_on“>input [i]  ♦  l_on->weight8 [j] [i] ; 

> 

y  «  y  +  l_on->theta[j] ; 
return  y; 

} 

*  Modified  Algorithm  for  ETAHN  Chip  Simulation 

float  fizy(y,hardne88) 
float  y,  hardne88  ; 

float  zz; 
int  i; 

hardnes8  ^  1.74; 

if  (y  >  8.0)  return  .89; 

if  (y  <  -8.0)  return  -.94; 

retumC  1. 83/(1. 0  +  (float)ezp(-(double)(hardne88  ♦  y)))  -  .94); 
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} 

delyCy.doft. del, output)  /«  computer  error  vector  for  output  *! 
float  *y,*doft,4idel;  /*  Interface  requires  computed  output  *! 
int  output;  /*  Desired  output  and  storage  for  error:  del. 

< 

int  i; 

if (parameters . output .data  i) 

loopi (output)  delCi]  =  0.1*(doft[i]-y[i]); 
else 

/♦♦♦♦  Derivative  of  ETANN  transfer  function  ♦♦♦♦/ 
loopi(output)  delCi]  =  0.95*(.94  +  y[i])*(.89  -  yCi])*(doft[i]-y[i]); 

} 

dely_linear(y,doft, del, output )/*computer  error  vector  for  output*/ 
float  *y,*doft ,*del;/*  Interface  requires  computed  output  */ 
int  output;  /*  Desired  output  and  storage  for  error:  del.  */ 

{ 

int  i; 

loopi(output)  del[i]  =  (doft[i]-y[i])/(float)output; 

} 

delz (z , del , del.dovn , v , upper , lover) 
float  *z,*del,*del_dovn,**w; 
int  upper, lover; 

< 

float  sum; 
int  i,j; 
loop j (lover) { 
sum  s  0.0; 
loopi (upper ){ 

sum  ■  sum  +  delCi]  *  vCi][j]; 

} 

/«*««  Derivative  of  ETANN  transfer  function  ****/ 
sum  ■  0.95*(.94  +  z[j])*(.89  -  x[j])  *  sum; 
del.dovn [j]  *  sum; 

} 

> 

check.linear(){ 

if (parameters. output. data  1) 

< 

net->layerl [parameters. layers-l]->propagate  »  calcy.linear; 

} 

) 
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Appendix  G.  ETANN  Characterization  Tools 
G.  1  Introduction 

The  C  programs  in  this  appendix  are  only  a  sampling  of  the  several  programs 
written  to  perform  sigmoidal  characterization  of  ETANN  neurons.  The  general  test 
procedures  were  as  follows:  1)  run  test  programs  to  get  data  from  eight  ETANN 
chips  while  varying  the  number  of  inputs  to  a  single  neuron;  2)  separate  the  data 
using  the  separate  data  program;  3)  compute  the  average  of  eight  chips  using  the 
averaging  data  program;  4)  calculate  the  MSE  versus  hardness  parameter  for  the 
collected  data;  and  finally,  5)  characterize  the  sigmoidal  transfer  function. 


G.2  C  Program  to  Collect  Data  from  ETANN 


*  Program:  Test  to  characterize  ETANN  transfer  function  using  2 
inputs  into  one  neuron. 


* 


*  Date:  8  July  1992 

♦ 

*  Author:  Capt  James  Calvin 

tinclude  "tsil.h" 
#include<stdio . h> 
#include<math . h> 


mainO 

•C 

REALS  weight.valueCSYNAPSES.PER.NEURON] ; 

REALS  bias_valueCBIASES_PER_NEURON] ; 

REALS  input.valueCNEURONS.PER.ARRAY]; 

REALS  etann.output [NEURONS.PER.ARRAY] ; 

REALS  actual.bias [ACTUAL.BIASES.PER.NEURON] ; 

REALS  etann_veight[SYNAPSES_PER.N£URON] ; 

REALS  8igmoid_gain  «  3.30; 

REALS  data  =  1.50; 

REALS  sum  [2]  ; 

static  REALS  test. weight [11]  =  {-2.5,  -2.0,  -1.5,  -1.0,  -.5,  0.0, 
.5,  1.0,  1.5,  2.0,  2.5,}; 

static  REALS  testl.input [9]  »  {-1.0,  -0.75,  -.5,  -.25,  0.0,  .25, 
.5,  .75,  1.0); 
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static  REALS  test2_input [9]  =  {-1.0,  -0.75,  -.5,  -.25,  0.0,  .25, 
.5,  .75,  1.0}; 

FILE  *out_ptr; 
int  w,x,y,2; 

Status!  status; 

out_ptr  =  fopen("test2.out'',''w"); 

*  Initialize  inputs  and  weights  to  zero 

forCx  =  0;  X  <  NEURONS.PER.ARRAY;  x++) 
input.value [x]  =  0,0; 
forCx  =  0;  X  <  SYNAPSES.PER.NEURON;  x-h-) 
we ight .value [x]  =  0.0; 

/  *  #  itt  4c  4c  %  *  #  ^  4i  *  ^  41  ^  4i  4i  %  #  #  :ti «  4i  *  #  #  i(E  i|c  III  *  4e  #  * 

*  Connect  the  ETANN  Chip 

4t4i4E4i4<*4<4t**4i4>4i4t4E4>***4i4!4c4t4c4t4t*4t4!4t4t4c4i4(>lt***4c4i/ 

printf("\n  connecting  etann"); 

if  (connect.etannO  !=0K){ 

printf("\n  error  in  connecting”); 
exit(O) ; 

} 

else 

printf(”\n  successful  connection”); 

/4e4>4t4>4i«4>4>4<4>4>««>)^**4>4c4>4>4t4>4c4c4c4E4>4t*4t4c4c4c*«4»tc4i 

*  Set  Sigmoid  Gain 

4t>|E4>4>4<4>4>*4<4i4i4<*4i4i4i4i4>4i4c4<4c4i4<4<4t4c4c4‘4i4»ii>^4>4>4t*4‘/ 

set.sigmoid.gainCsigmoid.gain) ; 

/4c4<«*««>(>*4i*4>4>*4t*4c4E*4>4<4i4i4t4c»4t4c«4i4^4c 

*  Set  Voltage  References 

4c4t«4i4t**4c«*«4>4<«)|>*4>4i4^4t4«l‘«*«4>*4i«*4>/ 
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set.voltage.ref erence.input (data) ; 
set .volt age.reference.output (data); 

/*wi***m************************* 

*  Begin  Test 

^^Jt^Hf^^^^tif^iti********************/ 

printfC'Xn  Running  Test*'); 
for  (w  =  0;  w  <  11;  w»-+){ 

veight.value [0]  =  test.weight [w] ; 
weight  .valued]  =  test.weight  [w]  ; 
set.weights (0,0, 2 , weight. value) ; 
get .weights (0,0,2, hweight. value [0] ) ; 
for  (y  =  0;  y  <  9;  y++){ 
input.value [0]  =  test 1. input [y] ; 
input.value [1]  =  test2.input [y] ; 

write.neuron.inputs(0,2,kinput. value [0] ) ; 
read.neuron.outputs(0, 1 ,ftetann.output [0] ) ; 
for  (y  =0;  y  <  2;  y++) 

sumCO]  +®  weight .value [y]  ♦  input.value [y]  ; 
fprintf  (out.ptr ,  ''%f  Xf  \n''  .sumCO]  ,etann_output  [0]  ) ; 
} 

fprintf  (out.ptr, ''\n") ; 

} 

*  Disconnect  ETANN 

printf("\n  Disconnect  ETANN'*); 
disconnect.etannO ; 

*  Close  Files 

«««*««*4c««***4>***>I<****4>4i***««*/ 

f close  (out.ptr); 
f close  (outl.ptr); 

} 


114 


G.S  C  Program  to  Separate  Collected  Data 

*  Program:  Separate  collected  data. 

>K  Date:  8  July  1992 

*  Author:  Capt  James  Calvin 

/it:^^**^tiit0i^iti**iitf^imm*}tt***^tm**********^t**‘***************************/ 

tinclude  <8tdio.h> 
finclude  <math.h> 

mainO 

< 


FILE  ♦in_ptr; 

FILE  ♦outl.ptr,  ♦out2_ptr,  *out3_ptr,  ♦out4_ptr,  ♦outS.ptr, 
♦oute.ptr,  *out7_ptr,  ♦out8_ptr,  ♦out9_ptr,  ♦outlO.ptr, 
*outll_ptr; 
int  z; 


float  act_l[10],  act_2[10],  act^SClO],  act .4  [10], 
float  act_6[10],  act .7  [10],  act_8[10],  act_9[10], 
act_ll[10] ; 

float  out_l[10],  out .2  [10],  out_3[10],  out .4  [10], 
float  out_6[10],  out .7  [10],  out_8[10],  out.9[10], 
out_ll [10] ; 


act .5  [10] ; 
act.l0[10], 

out .5 [10] ; 
out  ..10  [10] , 


in_ptr  =  fopen("chiph02.dat'',"r'*); 
if  (in_ptr**NULL) 
printf ("error  in  file  1"); 

outl.ptr  =  fopen("hl.dat”,"w”); 
if  (outl_ptr**N0LL) 

printf ("error  in  outfile”); 

out2„ptr  =  fopen("h2.dat”,”w"); 
if  (out2_ptr»*NULL) 

printf ("error  in  outfile”); 

out3.ptr  »  fopen("h3.dat",”ir”); 
if  (out3_ptr»=K0LL) 

printf ("error  in  outfile”); 
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out4_ptr  «  fopen(“h4.dat'*,"ir"); 
if  (out4_ptr*=NULL) 

printf ("error  in  outfile"); 

outS.ptr  =  fopen("h5.dat","w"); 
if  (out5_ptr==NULL) 

printf ("error  in  outfile"); 

out6_ptr  =  fopen("h6,dat","w") ; 
if  (out6_ptr==NULL) 

printf ("error  in  outfile"); 

out7_ptr  =  fopen("h7,dat","w"); 
if  (out7_ptr==NULL) 

printf ("error  in  outfile"); 

out8_ptr  =  fopen("h8.dat","w"); 
if  (out8_ptr==NULL) 

printf ("error  in  outfile"); 

out9_ptr  -  fopen("h9.dat","w"); 
if  (out9_ptr»*NULL) 

printf ("error  in  outfile"); 

outlO.ptr  =  fopen("hl0.dat","w"); 
if  (outlO_ptr==NULL) 

printf ("error  in  outfile"); 

outll.ptr  =  fopen("hll.dat","w"); 
if  (outll_ptr*=NULL) 

printf ("error  in  outfile"); 

for  (x  =  0;  X  <  9;  x++) 

fscanf (in_ptr,"%f  y,f  \n",  tact_l[x],  4out_l[x]); 
fscanf (in_ptr,"\n") ; 
for  (x  «  0;  X  <  9;  x-h-) 

fscanf (in_ptr,"%f  %f  \n",  kact.2[x],  kout.2[x]); 
fscanf (in_ptr,"\n") ; 
for  (x  «  0;  X  <  9;  x++) 

fscanf (in_ptr,"%f  %f  \n",  kact.SCx],  4out_3[x]); 
fscanf (in_ptr,"\n") ; 
for  (x  *  0;  X  <  9;  x++) 
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fscanf (in_ptr,"%f  %f  \n'',  kact_4[x3,  Jkout_4[x]): 
fscanf  (in_ptr,''\n'') ; 
for  (x  =  0;  X  <  9;  x++) 

fscanf (in_ptr,"Jlf  %f  \n'',  kact_5[x],  tout_5[x]); 
fscanf  (in_ptr,''\n") ; 
for  (x  =  0;  X  <  9;  x++) 

fscanf (in_ptr,"5lf  Xf  \n",  kact_6[x],  kout_6[x]); 
fscanf  (in_ptr,"\n'') ; 
for  (x  =  0;  X  <  9;  x++) 

fscanf  ( in.ptr ,  "7.f  */«f  \n" ,  kact_7  [x]  ,  kout_7  [x]  ) ; 
f scanf (in.ptr , "\n") ; 
for  (x  =  0;  X  <  9;  x++) 

fscanf (in_ptr,"7.f  */d  \n",  kact_8[x],  kout_8[x]); 
fscanf  (in.ptr,  ''\n") ; 
for  (x  =  0;  X  <  9;  x++) 

fscanf  (in.ptr,  "7.f  %f  \n",  kact.9[x],  Jtout.9[x]); 
fscanf (in.pt r,"\n") ; 
for  (x  =  0;  X  <  9;  x++) 

fscanf ( in.ptr, "7tf  7.f  \n",  kact.l0[x],  kout.l0[x]); 
fscanf (in.ptr, "\n") ; 

for  (x  *  0;  X  <  9;  x++) 

fscanf  (in.ptr, ''%f  7.f  \n'',  kact.ll[x],  kout.llCx]); 
for  (x  =  0;  X  <  9;  x++){ 

fprintf(outl.ptr,‘'7.f  \n'',  act_l[x],  out.lW); 
fprintf (out2.pt r,"7tf  7.f  \n*',  act_2[x],  out.2[x]); 
fprintf (out3.ptr,"7.f  5if  \n”,  act.3[x],  out.3[x]); 

fprintf (out4.ptr,''5lf  *t,f  \n",  act.4[x],  out_4[x]),* 

fprintf (out5.ptr,"7.f  Vd  \n”,  act.5[x],  out.5[x]); 

fprintf  (oute.ptr ,  ''7.f  7tf  \n'' ,  act .6  [x] ,  out .6  [x]  )  *, 

fprintf (out7.ptr,"%f  5#f  \n”,  act.7[x],  out.7[x]); 

fprintf (out8.ptr,''7.f  %f  \n'',  act_8[x],  out.8[x]); 

fprintf  (out9.ptr ,  "Jif  *4f  \n" ,  act.9  [x]  ,  out. 9  [x]  ) ; 

fprintf  (out  10.ptr,"5lf  %f  \n'',  act.lO[x],  out.lOCx]); 
fprintf (outll.ptr , "Xf  Xf  \n” ,  act.ll [x] ,  out.ll [x] ) ; 


f close  (outl.ptr); 
f close  (out2.ptr); 
f close  (out3.ptr); 
f close  (out4.ptr); 
f close  (outS.ptr); 
f close  (out6.ptr) ; 
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f close  (out7_ptr); 
f close  (outS.ptr); 
f close  (out9_ptr); 
fclose  (outlO.ptr) 
fclose  (outll.ptr) 
fclose  (in.ptr); 

}/♦  end  main  */ 


G.4  C  Program  to  Calculate  Average  Output  of  8  Chips 

/^^^^i^^^^H^^^^^^^^*^|^i^J^^:t^******0*********************************^C******/ 

/*  Program:  Calculate  the  average  output  of  8  ETANK  chips. 

/♦  Date:  7  July  1992 
/*  Author:  Capt  James  Calvin 

/^i^t*4tWL***m***************************************<¥******’¥*^>i^***/ 

tinclude  <stdio.h> 
f include  <math.h> 

mainO 

i 

FILE  *inl_ptr,  ♦in2_ptr,  *in3_ptr,  ♦in4_ptr; 

FILE  vinS.ptr,  ♦in6_ptr,  ♦in7_ptr,  ♦inS.ptr,  *outl_ptr, 
♦out2_ptr; 
int  z  s  0; 

float  act_avg[10] ,  act_lClO],  act_2[10],  act_3[10]; 

float  act.4ClO],  act_5[10],  act_6Cl0],  act_7[10],  act_8[10]; 

float  out_avg[10],  out.lClO],  out_2[10],  out .3  [10],  out_4[10]; 
float  out_5ClO],  out  .6  [10],  out .7  [10],  ottt_8[10],  hardness  [10] ; 

float  actl,  act2,  act3,  act4,  acts,  act6,  act7,  act8; 
float  outl,  out2,  out3,  out4,  outS,  out6,  out7,  out8; 

inl_ptr  =  fopen(''a3.dat'',''r'') ; 
if  (ittl.ptr=sNULL) 
pr int f ("error  in  file  1"); 

in2_ptr  =  fopen("b3.dat","r") ; 
if  (in2_ptr*=NDLL) 
printf ("error  in  file  2"); 

in3_ptr  *  fopen("c3.dat","r") ; 
if  (in3_ptr*»HDLL) 
printf ("error  in  file  3"); 

in4.ptr  =  fopen("d3.dat","r"); ^ 
if  (in4_ptr*»NULL) 
printf ("error  in  file  4"); 
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inS.ptr  «  fopen(''e3.dat",''r''); 
if  (in5_ptr*=NULL) 
printf ("error  in  file  5"); 

in6_ptr  ■  fopen("f3.dat","r") ; 
if  (in6_ptr==NULL) 
printf ("error  in  file  6"); 

in7_ptr  «  fopen("g3.dat","r") ; 
if  (in7_ptr*=NULL) 
printf ("error  in  file  7"); 

in8_ptr  =  fopen("h3.dat","r") ; 
if  (in8_ptr==NULL) 
printf ("error  in  file  8"); 

outl_ptr  =  fopen("avg3.dat","w") ; 
if  (outl.ptr—NULL) 
printf ("error  in  outfile"); 

out2_ptr  »  fopen("hard.dat","a"); 
if  (out2_ptr=NULL) 
printf ("error  in  hardness  outfile"); 


for  (x  ®  0;  X  <  9;x++){ 

fscanf (inl_ptr,"Xf  %f  \n",  hact.lCx],  hout.l[x]); 
printf ("Values  from  file  one:  %f  Xf  \n",  act_l[x], 
out_l [x] ) ; 

f  scanf  ( in2_ptr ,  "Xf  Xf  \n" ,  tact_2  [x]  ,  Jtout_2  [x]  ) ; 
printf ("Values  from  file  tvo:  Xf  Xf  \n",  act.2[x], 
out_2[x]); 

f scanf ( in3_ptr , "Xf  Xf  \n" ,  hact_3 [x] ,  Jfcout.3 [x] ) ; 
printf ("Values  from  file  three:  Xf  Xf  \n",  act_3[x], 
out_3[x]); 

fscanf (in4_ptr,"Xf  Xf  \n",  tact.4[x],  l:out_4[x]); 
printf ("Values  from  file  four:  Xf  Xf  \n",  act.4[x], 
out_4[x]); 

fscanf (in5_ptr,"Xf  Xf  \n",  tact.SCx],  Aont.5[x]); 
printf ("Values  from  file  five:  Xf  Xf  \n",  act_5[x], 
out_6[x)); 

fscanf (in6.ptr,"Xf  Xf">  tact.6[x],  feout_6Cx]); 
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printf ("Values  from  file  six:  %f  %f  \n",  act_6Cz], 
out .6 [x] ) ; 

f scanf (in7_ptr , "%f  %f " ,  ftact_7 [x] ,  tout_7 [x] ) ; 
printf ("Values  from  file  seven:  Xf  \n",  act_7[x], 

out«7 [x] ) ; 

f scanf (inS.ptr , "%f  Xf “ ,  tact_8 [x] ,  tout_8 [x] ) ; 
printf ("Values  from  file  eight:  %f  %f  \n",  act_8[x], 
out. 8 [x] ) ; 


for  (x  ®  0;x  <  9;x  ++){ 

act.avgCx]  =  (act_l[x]  +  act_2[x]  +  act_3[x]  +  act_4[x]  + 
act_5[x]  +  act.eCx]  +  act_7[x]  +  act_8[x])  / 

8.0; 

out_avg[x]  *  (out_l[x]  +  out_2[x]  +  out_3[x]  +  out_4[x]  + 
out_5[x]  +  out.6[x]  +  out.7[x]  +  out_8[x])  / 

8.0; 

} 

for  (x  »  0;x  <  9;x  ++){ 

f  printf  (out  l.ptr,"%.3f  5{.3f  \n",  act.avgCx]  ,  out.avgW); 
printf ("%.3f  y..3f  \n“,  act.avgCx],  out.avgCx]); 
if  (act.avgW  !*  O.OX 

hardness Cx]  *  -(float)log((double)((2/(out.avg[x]  +  D) 

-  D)  /  act.avgCx]; 

fprintf  (out2.ptr,"Xf  \n",  hardness [x] ) ; 
printf  ("Xf  \n",  hardness [x] ) ; 

} 

} 

f close  (inl.ptr); 
f close  (in2.ptr); 
f close  (inS.ptr); 
f close  (in4.ptr); 
f close  (inS.ptr); 
f close  (inS.ptr); 
f close  (in7.ptr); 
f close  (inS.ptr); 
f close  (outl.ptr); 
f close  (ottt2.ptr); 

}/*  end  main  */ 


121 


G.5  C  Program  to  Calculate  MSE  vs.  Hardness  Parameter 

/m***************^if********’*‘************>i‘**********<¥**^********* 

*  Program:  Program  to  calculate  the  total  MSE  vs.  hardness. 

*  Date:  9  July  1992 

*  Author:  Capt  James  Calvin 

iif^i^i,tt**********Dt***********************************************/ 

finclude  <stdio.h> 
finclude  <math.h> 

mainO 

FILE  ♦in_ptr,  ♦out.ptr; 
int  X.  y; 

float  act_l[10],  act_2[10],  act_3[10],  act_4[10],  act_5[10]; 
float  act_6[10],  act_7[10],  act_8[10],  act_9[10],  act_10[10], 
act_ll [10] ; 

float  out_l[10],  out_2[10],  out_3[10],  out_4[10],  out_5[10]; 
float  out. 6  [10],  out .7  [10],  out_8[10],  out.9[10],  out.l0[10], 
out_ll[10] : 

float  error  =0.0,  sum.error  =0.0,  sigmoid [90], 
hardness  =  0.00; 

/l(^l^1t^^i^i^^^Hi^^^iimntm****************************************^l********* 

*  Open  Files 

in.ptr  =  fopen("average.dat'',"r”) ; 
if  (in_ptr=NULL) 
printf ("error  in  file  1"); 
out.ptr  =  fopen("mse.dat","w") ; 
if  (out.ptr“NULL) 
printf ("error  in  outfile"); 

*  Read  in  Data 

for  (x  »  0;  X  <  9;  x++) 

fscanf (in.ptr, "%f  %f  \n",  Aact.l[x],  kout.l[x]): 
fscanf (in.ptr, "\n”) ; 


for 

(x  *  0;  X  <  9; 

X++) 

fscanf (in.ptr. 

••%f  Xf 

f  scanf  (in.ptr ,  ''\n' 

); 

for 

(x  »  0;  X  <  9; 

X++) 

fscanf (in.ptr. 

"Xf  Xf 

fscanf  (in_ptr,*’\n' 

); 

for 

(x  »  0;  X  <  9; 

X++) 

fscanf (in.ptr, 

•■7.f  Xf 

f  scanf  (  in.ptr ,  ''\n' 

): 

for 

(x  »  0;  X  <  9; 

X++) 

fscanf (in.ptr, 

"Xf  Xf 

fscanf (in_ptr,"\n' 

): 

for 

(x  =  0;  X  <  9; 

X++) 

fscanf (in.ptr, 

"7.f  7.f 

fscanf  (in_ptr,  ''\n' 

); 

for 

(x  =  0;  X  <  9; 

X++) 

fscanf (in.ptr, 

"Xf  Xf 

fscanf  (in_ptr,''\n' 

); 

for 

(x  =  0;  X  <  9; 

X++) 

fscanf (in.ptr, 

"Xf  Xf 

fscanf  (in.ptr, ''\n' 

); 

for 

(x  *  0;  X  <  9; 

X++) 

fscanf (in.ptr. 

"Xf  Xf 

f  s  canf ( in.pt r , ” \n ‘ 

); 

for 

(x  »  0;  X  <  9; 

X++) 

fscanf (in.ptr, 

"Xf  Xf 

f  scanf ( in_pt r , " \n ' 

); 

for 

(x  =  0;  X  <  9; 

X++) 

fscanf (in.ptr. 

"Xf  Xf 

Jbact_2[z],  iout_2[z]): 
kact_3  [x]  ,  Jtout_3  [x]  ) ; 
ftact_4 [x] ,  tout_4 [x] ) ; 
tact_5 [x] ,  kout_5 [x] ) ; 
Jtact_6  [x]  ,  ltout_6  [x]  )  ; 
4act_7 [x] ,  tout_7 [x] ) ; 
tact_8 [x] ,  tout.B [x] ) ; 
fcact.9 [x] ,  tout. 9 [xj ) ; 
tact.lO [x] ,  ftout.lO [x] ) ; 
tact.ll [x] ,  kout.ll [x] ) ; 


\n", 

\n". 

W. 

\n". 

\n”. 

\n". 

\n". 

V. 

\n'', 

V. 


*  Find  Total  NSE  vs.  Hardness  Factor 

^c*t******************************************^tl*****************/ 


for  (y  =  0;  y  <  90;  y++){ 
hardness  0.02; 
snB.error  s  o.O; 
for  (x  »  0;  X  <  9;  x++)-C 

sigmoidCx]  ^  i. 83/(1  (float)exp(‘*(double)hardness* 

act«l[x]))  -  0.94; 
error  ■  out_l[x]  -  sigmoid [x]; 
error  *■  error; 
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suiB.error  ♦*  error; 

} 

for  (x  =  0;  X  <  9;  x++){ 

sigmoid [x]  =  1. 83/(1  +  (float)exp(-(double)hardne8s* 
act_2[x]))  -  0.94; 
error  =  out_2[x]  -  sigmoid [x]; 
error  ♦=  error; 
sum. error  +*  error; 

} 

for  (x  =  0;  X  <  9;  x++){ 

sigmoid  [x]  =  1. 83/(1  +  (float)exp(-'(double)hardness* 
act_3[x]))  -  0.94; 
error  =  out_3[x]  -  sigmoid [x]; 
error  *=  error; 
sum_error  +=  error; 

} 

for  (x  =  0;  X  <  9;  x++){ 

sigmoidCx]  =  1. 83/(1  +  (float)exp(-(double)hardness* 
act.4[x3))  -  0.94; 
error  =  out.4Cx]  -  sigmoidCx]; 
error  ♦=  error; 

8ttm_error  error; 

} 

for  (x  =  0;  X  <  9;  x++){ 

sigmoidCx]  =  1. 83/(1  +  (float)exp(-(double)bardness* 
act.SCx]))  -  0.94; 
error  =  out.SCx]  -  sigmoidCx]; 
error  error; 
sum_error  +=  error; 

} 

for  (x  =  0;  X  <  9;  x++)-C 

sigmoidCx]  =  1. 83/(1  +  (float)exp(-(double)hardness* 
act_6Cx]))  -  0.94; 
error  *  out.GCx]  -  sigmoidCx]; 
error  ♦*  error; 
suffl_error  +=  error; 

} 

for  (x  =  0;  X  <  9;  X'H-){ 

sigmoidCx]  »  1. 83/(1  +  (float)exp(-(double)hardness* 
act_7Cx]))  -  0.94; 
error  «  out.TCx]  -  sigmoidCx]; 
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error  **  error; 
suB.error  ♦«  error; 

> 

for  (x  *  0;  X  <  9;  x++){ 

8igmoid[x]  *  1. 83/(1  +  (float) exp (-(double) hardness* 
act_8[x]))  -  0.94; 
error  *  out_8[x]  -  sigmoid [x]; 
error  **  error; 

8um_error  +=  error; 

} 

for  (x  «  0;  X  <  9;  x++){ 

sigmoidCx]  =  1. 83/(1  +  (float)exp(-(double)hardne88* 
act_9[x]))  -  0.94; 
error  *  out_9Cx]  -  sigmoidCx]; 
error  ♦=  error; 

8um_error  +=  error; 

} 

for  (x  =  0;  X  <  9;  x++){ 

sigmoidCx]  =  1.83/(l+(float)exp(-(double)hardness* 
act.lOCx]))  -  0.94; 
error  »  out.lOCx]  -  sigmoidCx]; 
error  **  error; 
saffl.erTor  error; 

} 

for  (x  =  0;  X  <  9;  x++)-( 

sigmoidCx]  =  1.83/(l+(float)exp(-(double)hardness* 
act.llCx]))  -  0.94; 
error  =  out.llCx]  -  sigmoidCx]; 
error  *=  error; 
sum_error  +=  error; 

} 

sttm_error  /=  99; 

fprintf  (out_ptr,'7.f  7.f  \n",  hardness,  sum.error); 


} 

f close  (in.ptr); 
f close  (out.ptr); 

}  /*  end  main  */ 
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Appendix  H.  C  Tools  for  INNTS 


H.l  Introduction 

This  appendix  contains  a  sampling  of  C  programs  used  in  research  for  inter¬ 
facing  the  Neural  Graphics  simulator  with  INNTS  software  and  hardware. 

H.2  C  Program  to  Write  Neural  Graphics’  Weights  to  ETANN 

*  Program:  Write  Neural  Graphics'  Weights  to  ETANN  Chip 

*  Date:  9  Aug  1992 

*  Author:  Capt  James  Calvin 

finclude  "tsil.h" 

#include<stdio . h> 

#include<math . h> 

mainO 

{ 

REALS  weight. value.layerl [16] ; 

REALS  weight _value_layer2 [20] ; 

REALS  bias.value.layerl [1] ; 

REALS  bias.value.layer2 [1] ; 

REALS  sigmoid.gain  =  3.30; 

REALS  data  «  1.50; 

FILE  *in_ptr,*out_ptr; 

0RD2  w,  z,  y,  z,  inputs,  outputs,  hidden; 
char  str [SO] ; 

StatusT  status; 

if  (Cin.ptr  =  fopen(''infile.wts",''r"))  —  NULL){ 
printf  ("I'm  not  able  to  open  the  input  file  \n"); 
exit(-l); 

} 

printf ("Enter  number  of  inputs,  a  space,  k  number  of 
outputs. \n") ; 

scanf("/(d  */d",  Ainputs,  Aoutputs); 

printf ("Enter  number  of  hidden  neurons. \n") ; 

scanf("%d",  Ahidden); 

fscanf  (in_ptr,"J(s  \n",  str); 
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/t^itiJt:**^^^****^^***************************** 

*  Connect  the  ETANN  Chip 

printf("\n  connecting  etann"); 

if  (connect.etannO  !=0  .){ 

printf(''\n  error  in  connecting"); 
exit(O); 

} 

else 

printf("\n  successful  connection"); 
set .et ann_gain (NORMAL.ETANN.GAIK) ; 
set.veight_precision(MAX_PRECISION) ; 

/^HlHtf:lf^f:itt*i^*mitt***********************n‘*** 

*  Set  Sigmoid  Gain 

*ilf**:t‘*********************************/ 

set.sigmoid.gainCsigmoid.gain) ; 
set.voltage.ref erence.input (data) ; 
set.voltage.ref erence.output (data) ; 

*  Reading  and  Writing  to  Layer  One 

^i****************************************/ 

for  (w  =  0;  w  <  hidden;  w++)-C 
for  (x  =  0;  X  <  inputs;  x++) 

fscanf  (in_ptr,  "Xlf",  kweight_value_layerl [x] ) ; 
fscanf  (in_ptr,  "5(lf  \n",  Abias_value_layerl  [0] ) ; 
set.etann.parametersCO, FEEDFORWARD) ; 
set .weights (v, 0 , inputs , weight .value.layerl) ; 
set .bias .weights (w, 0 , 1 .bias.value.layerl) ; 

} 

/t>fiti***i(fm’¥itf’*‘**it‘*********J¥****’¥********************************** 

*  Reading  and  Writing  to  Layer  Two 

fscanf  (in.ptr,"\n") ; 

for  (w  *  0;  w  <  outputs;  w++){ 
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for  (x  =  0;  X  <  hidden;  x++) 

fscanf  (in_ptr,''Xlf'',  *ireight_value_layer2[x]); 
f scant  (in_ptr,  "Xlf  \n",  ikbia8_value.layer2 [0] ) ; 
set.etann.parameters (0 ,  FEEDBACK) ; 
set.weight 8 (v , 0 , hidden , weight _value_layer2) ; 
set.bias.veights (v , 0 , 1 , bia8_value_layer2) ; 

} 

f close  (in.ptr); 

/*♦♦♦♦♦*♦*♦♦♦*♦♦**♦♦*♦♦*♦♦♦♦♦♦♦♦ 

*  Disconnect  ETANN 

****ttc********!^*}^*********^^***ilf*/ 

printfC'Xn  Disconnect  ETANN”); 
disconnect.etannC) ; 

printf  ("\n\n  Press  any  key  to  continue..."); 
getchO ; 

}  /♦  end  main  */ 
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H.S  C  Program  to  Write,  Read,  and  Classify  Input  Data  to  ETANN 

*  Program:  Write  Inputs  to  ETANN  Chip,  Read  Outputs,  and 

*■  Classify  the  Winner 

*  Date:  4  Aug  1992 

*  Author:  Capt  James  Calvin 

#include  "tsil.h" 

#include<stdio . h> 

#include<math . h> 

mainO 

{ 

REALS  in.dataCNEURONS.PER. ARRAY] ; 

REALS  out. data [NEURONS.PER.ARRAY]; 

REALS  sigmoid.gain  =  3.30; 

REALS  data  =  1.50; 

REALS  max; 

FILE  *in.ptr,»out.ptr; 

0RD2  V,  z,  y,  inputs,  outputs,  hidden; 

0RD2  countlCSO] ,count2C60] ; 

float  class.vinner,  actual.class ,  percentage,  junk; 
int  test  .vectors  *=  60,  total  =  0,  node,z; 

StatusT  status; 

if  ((in.ptr  =  fopen("infile'',”r’'))  ==  NULL){ 

printf  ("I’m  not  able  to  open  the  input  file  \n"); 
exit(-l) ; 

} 

if  ((out.ptr  =  fopen("outfile'',''w"))  ==  NULL){ 

printf  ("I’m  not  able  to  open  the  input  file  \n"); 
exit(-l) ; 

} 

printf ("Enter  number  of  inputs,  a  space,  t  number  of 
outputsAn”) ; 

scanfC'Xd  %d",  kinputs,  koutputs); 

printf ("Enter  number  of  hidden  neurons  An"); 

scanfC'^d",  khidden); 
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*  Connect  the  ETANN  Chip 

il::^t*7ln*W***************************tlf******/ 

printf("\n  connecting  etann*'); 

if  ( connect. etann ()  !sOK)-C 

printf("\n  error  in  connecting"); 
exit(O); 

} 

else 

printf("\n  successful  connection"); 
set_etann_gain(MORMAL_ETAKN_GAIM) ; 

/m************************************* 

*  Set  Sigmoid  Gain  k  Vref  out/in 

set_sigmoid_gain(sigmoid_gain) ; 
set .voltage.reference.input (data); 
set.voltage.ref erence.output (data) ; 

/**4i4i«4c*«4>*4>4i*«4t****«*4E*4^****ee«4^4>*4c***«* 

*  Begin  Writing  Inputs  to  ETANN 

****4(«*4i4c****««*4^**4i)|>«*«««««4>i»******«««*/ 

printf("\n  Writing  inputs"); 

*  Initialize  the  Count 

for  (x  «  1;  X  <*  outputs;  X'H-){ 
countl[x]  =  0; 
count2[x]  «  0; 

} 
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*  Writing  Inputs 


set. etann.parametersCO, FEEDFORWARD) : 
for  (x  =  0;  X  <  test.vectors;  x++){ 
fscanf  (in_ptr,"*/,f",  Ajunk); 
for  (y  =  0;  y  <  inputs;  y++) 

fscanf  (in_ptr,''5ilf'‘,  kin_data[y]  ) ; 
fscanf  (in_ptr,"*/f  \n",  Itactual.class) ; 
vrite.neuron.inputs (0 , inputs , in.data) ; 
clock.feedbackCl) ; 

read.neuron.outputs (0, outputs, kout_data[0] ) ; 
for  (z  =  outputs  -  1;  z  >=  0;  z — ){ 
w  =  outputs  -  z; 

fprintf  (out.ptr,"  Neuron  Output  for  class  y,d  =  )i.31f 
",w,  out.dataCz]); 

if  ((w  7,  2)  ==  0)  fprintf  (out.ptr, '’\n'') ; 

} 


*  Classify  the  Output 

/♦Neural  Graphics  calls  neuron  0  the  leist  class  (reverse  logic)*/ 
max  =  “1.0; 

for  (y  =  0;  y  <  outputs;  y++){ 
if  (out.dataCy]  >=  max)^ 
max  s  out.data[y] ; 
node  *  y; 

} 

} 

class.vinner  ~  (float) (outputs  “  node); 
fprintf  (out.ptr, "\n  Desired  Output  -  %.0f  Winner  =  %.0i 
\n\n",  actual.class ,  class.vinner); 
if  (actual.class  ^  class.vinner)  ■{ 
y  «  (int) class.vinner; 
countlCy]  +=  1; 
total++; 

> 

for  (y  =  1;  y  <=  outputs;  y++) 

if  ((int) actual.class  y)  count2Cy]  +=  1; 

} 


131 


/i|^^^*^^^^:ti**l|^:t^^i^i1|^***:^t^ltl*lt^**>l^**************************************** 

*  Print  the  Output 

fprintf  (out _ptr, "Total  Number  of  Test  Vectors  =  '/,d  \n", 
test.vectors) ; 

for  (x  *  1;  X  <=  outputs:  x++){ 

percentage  =  ((float) count! [x]  /  (float) count2 [x] ) ’•■100; 
fprintf  (out _ptr, "Class  Xd:  Total  =  Xd  Correct  =  y,d 

Percent  =  */,.2f  \n",x,count2[x]  .count! [x]  .percentage) : 

} 

fprintf  (out.ptr. "Total  Number  of  Classified  Right  =  */,d  \n\n". 
total) ; 

percentage  =  ((float)total  /  (float)test_vectors)*!00; 
printf  ("\n\n  Total  Test  Vectors  =  5(d".  test.vectors) ; 
printf  ("\n\n  Total  Classified  Right  =  Xd".  total); 
fprintf  (out _ptr. "Percentage  Right  =  y.2f  \n".  percentage); 
printf  ("\n\n  Percentage  Right  =  %.2f  ".  percentage); 

*  Disconnect  ETANN 

t****tHtlf********J¥***************/ 

printf ("\n  Disconnect  ETANN"); 
disconnect.etannO ; 
f close  (in.ptr); 
f close  (out.ptr) ; 

printf  ("\n\n  Press  any  key  to  continue..."); 
getchO; 

}  /*  end  main  ♦/ 
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H.4  C  Program  to  Write  Neural  Graphics  ’  Weights  to  EMB 

*  Program:  Write  Neural  Graphics'  Weights  to  EMB 

*  Date:  27  Aug  1992 

*  Author:  Capt  James  Calvin 

^^1^^^^^^Hf^H^***^^^^^^*^^^^*^H^^HH^****^^*^^^^^^H^****^^^H^***^^****^^******^^^^**^^*/ 

tinclude  “tsil.h" 

#include<stdio . h> 

#include<math . h> 

mainC) 

{ 

REALS  veight_value_layerlCl6] ; 

REALS  veight_value_layer2C35] ; 

REALS  bias.value.layerl [6] ; 

REALS  bias.value_layer2 [6] ; 

REALS  bias; 

FILE  ♦in_ptr,*out_ptr; 

0RD2  V,  z,  y,  z.  inputs,  outputs,  hidden; 
char  str[S0] ; 

StatusT  status; 

if  ((in.ptr  *  f open (" inf ile.wts”,''r’'))  =  NULLX 
printf  ("I'm  not  able  to  open  the  input  file  \n"); 
exit(-l) ; 

} 

printf ("Enter  number  of  inputs,  a  space,  k  number 
of  output s. \n" ) ; 

scanf("%d  %d",  tinputs,  Itoutputs); 

printf  ("Enter  number  of  hidden  neuronsAn"); 

scanf("%d",  Ahidden); 

fscanf  (in.ptr, "Xs  \n",  str); 

*  Connect  the  EMB 

printf ("\n  Place  your  EMB  in  the  adapter  socket,  then  press 
a  key. . 
getchO ; 
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if  (connect.emb(i80170.NX)  !=  0K){ 
printf("\n  error  in  connecting"); 
exit(O) ; 

} 

else 

printf("\n  successful  connection"); 
set .et ann.gain (NORMAL.ETAMN.GAIN) ; 
set_weight_precision(MAX_PRECISION) ; 

/ili*mitt**itt:ti:tt****************************J^*** 

*  Reading  and  Writing  to  Layer  One 

for  (w  =  0;  w  <  hidden;  w++){ 
for  (x  =  0;  X  <  inputs;  x++) 
fscanf  (in.ptr,  "7,lf",  tveight_value_layerl[x]) ; 
fscanf  (in.ptr,  "7lf  \n",  Abias); 
for  (x  =  0;  X  <  6;  x++) 
bias_value_layerl [x]  =  bias  /  6.0; 

set.etann.parameters (0 , FEEDFORWARD) ; 

set .weights (w, 0, inputs ,weight_value.layerl) ; 

set. bias .weight s (w ,0,6, bias.value.layerl) ; 

} 

*  Reading  and  Writing  to  Layer  Two 

ntit‘**************************m**********Jti*****7if*****m****m’ti*****/ 

fscanf  (in.ptr,"\n") ; 
for  (w  =  0;  w  <  outputs;  w++){ 
for  (x  =  0;  X  <  hidden;  x++) 
fscanf  (in.ptr , "Xlf " ,  Aweight.value.layer2 [x] ) ; 
fscanf  (in.ptr,  "7,lf  \n",  Abias); 
for  (x  *  0;  x  <  6;  x++) 
bia8_value_layer2 [x]  «  bias  /  6.0; 
set.etann.parameters (0 ,  FEEDBACK) ; 
set.weight 8 (w , 0 , hidden , weight_value.layer2) ; 
set.bias.weight s (w , 0 , 6 , bia8_yalue_layer2) ; 

} 

f close  (in.ptr); 
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/i)f,tt*******************’<f******’^** 

*  Disconnect  ETANN 

printf("\n  Disconnect  EMB"); 
disconnect.embO ; 

printf  ("\n\n  Press  any  key  to  continue...") 
getchO ; 

}  /♦  end  main  */ 
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H.5  C  Program  to  Write,  Read,  and  Classify  Input  Data  to  EMB 

*  Program:  Write  Inputs  to  EMB  and  Read  Outputs 

*  Date:  27  Aug  1992 

*  Author:  Capt  James  Calvin 

♦include  "tsil.h" 

#include<stdio . h> 

#include<math . h> 

mainO 

{ 

REALS  in_data[16]; 

REALS  out.dataCSO] ; 

REALS  max; 

FILE  *in_ptr,*out_ptr; 

0RD2  V,  z.  y.  inputs,  outputs,  hidden; 

0RD2  count 1 [31] , count2 [31] ; 

float  class.vinner.  actual.class ,  percentage,  junk; 
int  test.vectors  «  750.  total  =  0.  node.z; 

StatusT  status; 

if  ((in.ptr  =  f open ('* inf ile"."r''))  *=  NULL){ 

printf  ("I’m  not  able  to  open  the  input  file  \n"); 
exit(-l) ; 

} 

if  ((out.ptr  =  fopen("outfile".”w"))  =  NULL){ 

printf  ("I’m  not  able  to  open  the  input  file  \n"); 
exit(-l) ; 

} 

printf ("Enter  number  of  inputs,  a  space,  k  number  of 
output s.\u"); 

scanf("%d  %d".  kinputs.  koutputs); 

«  Connect  the  EMB 

printf ("\n  Place  your  EMB  in  the  adapter  socket,  then  press 
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a  key. . 
getchO; 

if  (connect_emb(i80170_NX)  !*  0K){ 
printfC'Xn  error  in  connecting*'); 
ezit(O) ; 

} 

else 

printf("\n  successful  connection"); 
set_etann_gain(MORMAL_ETANN_GAIN) ; 

/  #  4c  It!  4c  4c  4i  *  1|E  #  Iti  4e  4e  4i  *  )tt  4c  4c  4c  4i  *  *  Iti  41 

*  Initialize  the  Count 

4c  4c  4c  4c  4c  41 4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c  4c4</ 

for  (x  =  1;  X  <=  outputs;  x++){ 
count l[x]  =  0; 
count2[x]  ~  0; 

} 


/4c4c4c4:4c4e4c4c4c4c4c4c4c4c4c4>4c4c4c4>4>4c4c4c4c4c4c4c4c4c4c4!4c4c4c4c4c4c4c4c/ 

/«4c4c4c4c«4c4c4c«4c4c4c4c4c4c4c4ce4c4c4c44c4c4c4c4i4c4c4c4c4c4c4c4>4c4c4c*4c 

*  Writing  Inputs 

4c  4c  4c  4c  4c  4c  4c  4c  4c44c  4c  4>  4c  4c  4c  4c  4c  4c  4c  4c  4c  4>  4c  4ci4  4c  4c  4c  4c  4c  4c  4c  4c  4c  4>  4c  4>4>  4c  4c/ 

printf("\n  Writing  inputs"); 
set.etann.parametersCO, FEEDFORWARD) ; 
for  (x  =  0;  X  <  test  .vectors ;  x++)-C 
fscanf  (in_ptr,"JCf",  kjunk); 
for  (y  =  0;  y  <  inputs;  y++) 
fscanf  (in_ptr,"%lf",  kin.data[y]); 

fscanf  (in.ptr,"%f  \n",  kactual.class) ; 
vr ite.neuron.input s (0 , inputs , in.data) ; 
clock_f eedback(l) ; 

read.neuron.outputs (0, outputs, kottt.dat a [0] ) ; 
for  (z  ■  outputs  -  1;  z  >*  0;  z — ){ 
w  «  outputs  -  z; 

fprintf  (out.ptr,"  Neuron  Output  for  class  %d  « 
%.31f  ",v,  out_data[z]); 
if  ((w  %  2)  ««  0)  fprintf  (out_ptr,"\n"); 

} 
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/Hitti)lf1f:^^i^:tt***********tli*******************0**i¥********************* 

*  Classify  the  Output 

:it*m***mm**********************************m*********************/ 

/*  Neural  Graphics  calls  neuron  0  highest  class  (reverse  order)*/ 
max  =  -1.0; 

for  (y  =  0;  y  <  outputs;  y++){ 
if  (out_data[y]  >=  maz){ 
max  -  out_data[y]; 
node  >=  y; 

} 

} 

class.vinner  >  (float) (outputs  -  node); 

fprintf  (out_ptr,"\n  Desired  Output  =  7,. Of  Winner  = 

7,. Of  \n\n", 

actual.class,  class.vinner) ; 
if  (actual.class  -=  class_vinner){ 
y  *  (int)class_winner; 
count l[y]  +=  1; 
total-M-; 

} 

for  (y  =  1;  y  <=  outputs;  y++) 
if  ((int)actual.class  =*  y)  count2[y]  1; 

} 

/*>tniti^i*nt****it>**********************m****************************m* 

*  Print  the  Output 

}itnfjti*********************************************************itt***/ 

fprintf  (out _ptr, "Total  Number  of  Test  Vectors  ®  %d  \n", 
test.vectors) ; 

for  (x  =  1;  X  <=  outputs;  x++)-( 

percentage  =  ((float) countlCx]  /  (float )count2[x]) *100; 
fprintf  (out _ptr, "Class  7d;  Total  =  7d  Correct  =  7d 

Percent  =  X.2f  \n",x.cottnt2[x] , countlCx] .percentage) ; 

} 

fprintf  (out .ptr. "Total  Number  of  Classified  Right  « 

\n\n",  total); 

percentage  »  ((float)total  /  (float)test_vectors)*100; 
printf  ("\n\n  Total  Test  Vectors  »  7d",  test.vectors); 
printf  ("\n\n  Total  Classified  Right  »  Xd",  total); 
fprintf  (out _ptr, "Percentage  Right  »  %.2f  \n",  percentage); 
printf  ("\n\n  Percentage  Right  •  %.2f  ”,  percentage); 
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/itt**J¥*************************** 

*  Disconnect  ENB 

printfC'Xn  Disconnect  EMB"); 
disconnect.embO ; 
f close  (in.ptr); 
f close  (out.ptr); 

printf  ("\n\n  Press  any  key  to  continue...”) 
getchO; 

} 
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